Creating a Cognitive Tutor - CMUCTAT/CTAT GitHub Wiki

Creating a Cognitive Tutor for fraction addition

Table of Contents

Note: Alternative Tool Set

  1. The CTAT Jess Tools
  2. Examining Working Memory
  3. Examining Templates, Facts, and Production Rules in Eclipse
    1. Working Memory types (templates)
    2. Fact assertions, slot assignments
    3. Production rules
  4. The Conflict Tree
  5. Create a new problem
  6. Writing a production rule for a new strategy
    1. Write the rule
    2. Pseudo-code/English version
    3. Jess code version
  7. Evaluate and Test Rule
  8. Why Not
  9. Conclusion

Alternative Tool Set

This tutorial describes how to create and modify production rules for a fraction addition tutor with the CTAT Jess tools and the fraction addition materials supplied with CTAT. However, since the release of CTAT 4.4, we now recommend using the JavaScript Model Tracer instead of Jess. Jess tutors rely on a server-based Java-language model tracer, which is supported in the Authoring Tool environment and the Java Tutoring Service. The JavaScript-based model tracer, however, needs no server-based model tracer, so its use is not limited by server capacity. The Nools production rule language differs from Jess, but the overall concepts for building cognitive tutors are the same, and many of the authoring tools described here are available in the browser developer tools' console for JavaScript.

Prerequisites:

  • CTAT 4.2+
  • The Fraction Addition example containing interface, BRD, and Jess files.
    • Copy the Jess example FractionAddition tutor to your CTAT workspace folder:

      • If you have CTAT 4.3 or later installed, copy the HTML FractionAddition folder in the Cognitive Tutor Authoring Tools\Projects\ directory of your installation to CTAT\FractionAddition,

        -or-

      • Download the example: FractionAddition.zip, and unzip the downloaded file into your CTAT workspace.

    Note: If you have built your own Fraction Addition interface by following the steps in Building an HTML student interface for fraction addition, you can use your own interface instead of the one included in the example.

  • (Optional) Eclipse with Jess plugin for editing production rules. Eclipse should be configured according to the configuration instructions.

The CTAT Jess Tools

CTAT provides a suite of Jess tools for working with a cognitive model and tutor. In this portion of the tutorial, we'll examine a complete model for fraction addition using the following tools:

  • Working Memory Editor
  • Conflict Tree
  • Why Not?
  • Eclipse

Let's start by launching these tools, as well as the Eclipse environment:

  1. Launch CTAT for HTML and Flash.

  2. Set the Tutor Type to Cognitive Tutor (Jess). All Jess windows will open. (If you don't see Jess tool windows, select Window > Restore Default View.)

    jess tools

    Figure 1. CTAT Jess Tools

  3. Open the student interface by clicking File > Launch HTML Interface... and select the FractionAddition.html file from the HTML directory in your package folder.

  4. Open the behavior graph "1-3plus2-3.brd" from your package's FinalBRDs folder by clicking File > Open Graph.

  5. Check the Cognitive Model Folder setting in the WME Editor window. A tool tip displaying the full path of the current folder is shown when you mouse over the "Cognitive Model Folder:" text at the top of the WME Editor window.

    Note: By default, the Cognitive Model Folder is set for you whenever you open a behavior graph; CTAT looks for a "CognitiveModel" folder in your package. This is the folder containing the production rules file (productionRules.pr), working memory templates (wmeTypes.clp), and the working memory file for the behavior graph you are using (1-3plus2-3.wme).

  6. Launch Eclipse if you've configured it to use the Jess Editor plug in.

Examining working memory

The Working Memory Editor window is one of the Jess tools available in CTAT. It can be used to inspect and modify the contents of the cognitive model's working memory, or a collection of facts.

When you open graph "1-3plus2-3.brd" while in Jess mode, the cognitive model files are loaded and working memory is populated for this problem. You should see a few messages in the Jess Console window reporting that the Jess files were loading successfully:

*****Engine Cleared*****

Reading wmeTypes.clp (C:\Users\user1\CTAT\FractionAddition\CognitiveModel\wmeTypes.clp)
Reading productionRules.pr (C:\Users\user1\CTAT\FractionAddition\CognitiveModel\productionRules.pr)
Reading 1-3plus2-3.wme (C:\Users\user1\CTAT\FractionAddition\CognitiveModel\1-3plus2-3.wme)

Note that file "1-3plus2-3.wme" was loaded because it corresponds to the name of the behavior graph you opened.

The Working Memory Editor window shows the "facts", or contents of working memory, for the current problem. You'll notice that some of these facts mirror the set of components in the student interface. Why would working memory mirror the interface? The structure of a problem often (although not always) follows the interface. This concept should become clearer as you work through the tutorial.

In working memory, there's always a key fact called problem, and all the other facts are linked to that fact. For example, the fact problem has a slot called given-fractions which contains facts corresponding to the CTATTextInput components in the interface. The facts for the CTATTextInput components are represented in working memory by the textField template and facts.

working memory editor

Figure 2. Working Memory Editor with fraction template and facts shown

But CTATTextInput components alone are not useful in a cognitive model. Much more useful is the concept of a fraction, a collection of textField facts in a unit that represents a fraction. To create fractions, we've extended working memory to include a template type of fraction. You can view the current problem's fractions by expanding the fraction template, and clicking on the various fraction facts in the top panel (see Figure 2). These facts contain references to the components in the interface. To find a particular value, we need to follow these references to the CTATTextInput component.

For example, since the start state for the problem 1/3 + 2/3 has 1, 3, 2, and 3 entered in the CTATTextInput components of the interface, you can view these values in the Working Memory Editor by clicking on the textField template, and the facts within it (e.g., firstNumGiven, firstDenGiven). These facts are referenced from the various fraction facts: within the fraction template, click on given-fraction1; its numerator slot has a value of "<Fact-4>", which maps to the firstNumGiven interface component (which in turn holds the value of "1").

Tip: A right-click (Windows) or CTRL+click (Mac) on a fact in a slot value takes you to that fact in the Working Memory Editor. Pressing CTRL-[ takes you back to the previous fact.

You can do quite a lot with the Working Memory Editor besides just viewing the facts. From here, you could edit the start state and save it (useful when the start state the tools generate for you is not sufficient).

Examining Templates, Facts, and Production Rules in Eclipse

The Working Memory Editor visualizes working memory and makes the facts it contains easily accessible. But where is this initial set of data stored? And what if we want to do a larger, more powerful overhaul to working memory or the fact types?

Note: If you are not using the Jess Editor in Eclipse, you can use a plain text editor instead. Editing Jess files in Eclipse is recommended but not required.

Working Memory types (templates)

The types, or templates, for facts are stored in the file wmeTypes.clp, which can be generated from CTAT during problem authoring. We can open the file in Eclipse to see its contents. For fraction addition, this file was modified to include new templates such as add-fractions-goal, convert-fraction-goal, and fraction. The type problem was also amended to consist of given-fractions, converted-fractions, and answer-fractions. We refer to this process as augmenting working memory.

In Eclipse, all deftemplate statements are collapsed by default. You expand each statement by clicking the "+" icon next to the start of each deftemplate statement.

Fact assertions, slot assignments

The initial set of variables and facts used in working memory is defined in a file called {problem-name}.wme. This file must have the same problem name as the BRD file that accompanies it.

Open the file in the Eclipse IDE to see its contents. While covering the syntax of the Jess language is beyond the scope of this tutorial, you should note the fact assertions for the various fraction types, and the slot assignments for each.

Note: If you are not using the Jess Editor in Eclipse, you can use a plain text editor instead. Editing Jess files in Eclipse is recommended but not required.

Like wmeTypes.clp, the .wme file can be generated by CTAT to aid in cognitive modeling. As we will see later, modifying the list of facts that will appear in working memory is also useful.

Production rules

Now that we've explored the two main Jess files for supporting working memory, we can delve into the meat of the cognitive model: the production rules file.

Production rules files in CTAT tutors have the name productionRules.pr, and must reside in the same directory as the rest of the cognitive model files. You can open a production rules file by selecting it from within the Eclipse IDE.

Tip:

If you don't see a file in Eclipse, try refreshing Eclipse's view: select the project in Eclipse that contains the CTAT files and press F5 on the keyboard.

The production rules for fraction addition are each prefaced by the name of the rule, and an English version of the Jess production rule. The English version is pseudo-code, or an algorithmic description of what the rule does. This English version has two parts: IF and THEN. IF corresponds to the left-hand side (LHS) of the rule; THEN corresponds to the right-hand side (RHS). Good English versions of rules help not only during authoring and debugging, but when someone who's not familiar with the rule tries to understand it.

We'll revisit the production rules file after experimenting with the fraction addition tutor.

The Conflict Tree

When the student performs a step in the interface, the model-tracing algorithm searches for a sequence of rule activations that produce the same action as the student. This search is represented visually in the Conflict Tree.

The Conflict Tree can best be understood by demonstration. So let's enter a value into the interface, and examine the Conflict Tree:

  1. With the problem 1/3 + 2/3 open, set CTAT to Test Tutor mode. Also ensure that the Tutor Type is set to Cognitive Tutor (Jess).
  2. In the interface, enter the value "3" as the numerator of the answer fraction at the bottom.

CTAT Conflict Tree

Figure 3. The Conflict Tree after entering the numerator of the answer fraction.

The number 3 should turn green, since 3 is a valid numerator for the answer fraction. In the Conflict Tree, a number of rows should appear as shown in Figure 3.

We can see that the rule same-denominators fired first since it's the first item in the main Chain. But why did this rule match, and what were its effects? We can find out by clicking the rule same-denominators.

Clicking the rule same-denominators brings up a new colorful window called Why Not?. The green text above the Name/Value table in the middle tells us that the LHS of the rule matched. In the top half of the window, the rule definition is displayed, with variables highlighted. Hover your mouse cursor over the variable ?d. A tool-tip tells you that the value is 3. Lines 5 and 6 of the rule represent a pattern that tests if both denominators are the same value. (We'll cover the Why Not? window in more detail later.)

What was the effect of this rule's activation? We look at the right-hand side of the rule--the Jess code after the => delimiter--and see that a new sub-goal was created: add-fractions-goal. This goal is now visible in the Working Memory Editor under the template add-fractions-goal.

The chain of rule activations didn't stop there. As you can see in the Conflict Tree, the rule add-numerators also activated. Furthermore, three green checks appeared to the left of the rule name in the columns S, A, and I (Selection, Action, and Input). Clicking on any of the checks shows more information about the model during the past step: the rule's prediction and the student's entered value. According to the rule add-numerators, the model predicted an input of 3 on the component ansNum1, and the student (you, in this case) entered 3 into the same component.

Let's enter an incorrect value to see what the model does. Enter 4 as the denominator of the answer fraction. The value turns red, and the Conflict Tree updates with new information. Green checks appear to the left of the first copy-answer-denominator rule in the S and A columns, and a red X appears in the I column. If you click on any of these, you see that the model predicted the input 3, but we demonstrated 4, a clear mismatch.

Replace the incorrect denominator in the interface with the correct value, 3. Three green checks appear next to the rule copy-answer-denominator; the model matches the student input.

We can also see how hints are defined for a problem. Click the Hint button to see a hint. At this point in the tutor, if we click the name of the rule in the Conflict Tree, we see that the hint is defined in the RHS of the rule. (It even references a few variables.)

When you are done exploring the tools with this problem, you can close the interface and graph. Click "No" at the prompt to save the graph.

Tip:

You'll notice that links are added to the graph when you enter input in the student interface that matches a rule. A graph for a cognitive-model problem does not require links, but links in the graph can be useful when you are testing a cognitive model.

Create a new problem

We're now ready to create a new fraction addition problem. We'll create a new BRD for the problem 1/4 + 3/8, as well as a new WME file. To do so, we can use a .WME file from an existing problem as a template.

  1. In Windows Explorer or Finder, duplicate the file 1-4plus3-16.wme and save it in the same directory (\FractionAddition\CognitiveModel).

  2. Give the duplicate file the name 1-4plus3-8.wme.

  3. Open the new file in Eclipse.

  4. Modify ?var11, with name of secDenGiven, to have a value of 8:

    ; MAIN::textArea
(modify ?var11
    (name secDenGiven)
    (value 8)
)

  5. Save and close the file.

We're now able to create a new problem in CTAT:

  1. Click File > New > New Graph

  2. With the new Graph window selected, open the student interface again by selecting File > Launch HTML Interface.... (If you still have the interface open for problem 1/3 + 2/3, you can close it.)

  3. In the student interface, enter the fractions 1/4 and 3/8 as given fractions.

    Be sure to tab off of the last field, or it will not be recorded as part of the start state!

  4. In CTAT, click Graph > Create Start State.

  5. Enter the name 1-4plus3-8.

  6. Click File > Save Graph As and name the graph 1-4plus3-8.

You should now have a working problem for 1/4 + 3/8. You can test the model in Test Tutor mode to verify that it works.

Writing a production rule for a new strategy

We're now going to discover a weakness of the current model, and create a new production rule to address it.

  1. Convert the first fraction, 1/4, by entering 2 for the numerator and 8 for the denominator
  2. Skip to the answer fraction, and enter 5, the correct numerator, as shown in Figure 4. The model considers this input to be incorrect.

Figure 4: The current model requires all fractions to be converted, even if one already has the LCD as its denominator.

What's going on here? The current model has no understanding of fractions where one denominator is a multiple of the other. Rather, it understands 1/4 and 3/8 as two fractions that both need to be converted!

We will now create a new rule called one-denominator-multiple-of-other.

Write the rule

The first step in creating the rule one-denominator-multiple-of-other is to start with an existing similar rule as a template. The most similar rule in productionRules.pr is determine-lcd. In Eclipse, copy this entire rule to the space below it. Then modify the name of the rule to be our new rule one-denominator-multiple-of-other. Do this in both the comment at the top of the rule, and the line that starts (defrule .

Pseudo-code/English version

Next, we modify the English version of the rule to match the desired functionality of the new rule. We use the pattern-matching capabilities of Jess to define a succinct pseudo-code rule:

;; ONE-DENOMINATOR-MULTIPLE-OF-OTHER
;; IF there are no subgoals
;; and F1 is one of the given fractions
;; and F2 is the other given fraction
;; and D1 is the denominator of F1
;; and D2 is the denominator of F2
;; and D1 is a multiple of D2
;; THEN
;; Set a subgoal to convert F2 to denominator D1
;; Set a subgoal to add F1 and the converted fraction

The main differences are:

  • fractions are now variables;
  • salience of the rule is slightly higher to compensate for the overly general rule determine-lcd;
  • we test that the denominators are not equal to each other;
  • we test that one of the denominators divides evenly into the other with no remainder using the mod operation;
  • only one fraction must be converted; and
  • the two fractions to add are the fraction with the larger denominator and the new, converted fraction

Jess code version

You will now write the new Jess code for the rule by copy and paste.

    (defrule one-denominator-multiple-of-other
       (declare (salience 200))
       ?problem <- (problem
          (given-fractions $? ?f1 $?)
          (subgoals))
       ?problem <- (problem (given-fractions $? ?f2 $?))
       (test (not (eq ?f1 ?f2)))
       ?f1 <- (fraction (denominator ?denom1))
       ?f2 <- (fraction (denominator ?denom2)
           (has-converted-form ?conv))
       ?denom1 <- (textField (value ?d1&:(neq ?d1 nil)))
       ?denom2 <- (textField (value ?d2&:(neq ?d2 nil)))
       (test (= 0 (mod ?d1 ?d2)))
    =>
       (bind ?sub1 (assert (convert-fraction-goal  (fraction ?f2) 
          (denominator-value ?d1))))
       (bind ?sub2 (assert (add-fractions-goal (fractions ?f1 ?conv))))
       (modify ?problem (subgoals ?sub1 ?sub2))
       (construct-message
          "[ What denominator could you use to add the two fractions? ]"
          "[ Since " ?d1 " is a multiple of " ?d2 ", use " ?d1 " as the 
          denominator of the sum fraction.  ]"
          ))

Evaluate and Test Rule

Save your new rule by selecting File > Save in Eclipse.

Reload the modified production rules file by clicking Graph > Jump to Start State in CTAT. CTAT evaluates the validity of the file whenever you re-enter the start state of the graph.

With both the Conflict Tree and student interface visible, you are now ready to test your rule. As before, enter 2 as the numerator of the converted fraction. The input should turn green, but more importantly, the Conflict Tree should show that the one-denominator-multiple-of-other rule you just added was activated.

Continue solving the problem, pausing to examine the Conflict Tree after each step. Be sure to skip conversion of the second fraction, as this is the test case that produced an error in the earlier model.

Why Not?

Often times while testing your production rules, a particular rule that you expect to fire won't. In this case, we can invoke the Why Not? window to tell us more. In the previous section, we had a functioning model where all rules that we expected to fire did. Let's augment the model with a logical error.

In Eclipse, replace the modulus operator mod with /, the operator for division, in rule one-denominator-multiple-of-other. Save your production rules file and return to the start state.

Next, test the faulty model by entering 2/8 for the first converted fraction, and 5 as the numerator of the answer fraction. You'll notice that the input turns red, and the rule one-denominator-multiple-of-other did not fire. But why not?

To launch the Why Not? window, we first click the text Chain at the top of the Conflict Tree to show us a list of all production rules in the model. We then select one-denominator-multiple-of-other from this list (See Figure 5).

Figure 5. Asking Why Not? from the Conflict Tree

The Why Not? window appears with the following text in red:

In line 9: (= 0 (/ ?d1 ?d2)) did not return TRUE.

This means that the test on this line returned false, so the rule could not fully activate. This makes sense since the one denominator divided by the other equal denominator would yield 1, not 0. We can see that in this operation, the return value was FALSE; hovering over the highlighted portion of the rule in the top half of the window shows us this. This information is also displayed in the last line of the Why Not? window.

Why Not?

Figure 6. Why Not? one-denominator-multiple-of-other

Change your production rules back to use the mod operator. Test that your production rules work again by returning to the start state and evaluating the tutor in Test Tutor mode.

Conclusion

This concludes the Cognitive Tutor tutorial. If you'd like, you can expand the fraction addition model to account for different problem-solving strategies or student misconceptions.

Back to top