Example-tracing Tutors

Table of Contents

1. Overview
2. Demonstrate
3. Generalize
   1. Input Matching
   2. Student- and Tutor-Performed Actions
   3. Ordering Constraints
   4. Optional Steps and Minimum/Maximum Traversals
4. Annotate
   1. Hints
   2. Feedback
   3. Skills
5. Test

Overview

This section will walk you through the main phases of authoring an Example-tracing tutor with CTAT. For a full, example-based tutorial, see Creating an Example-tracing Tutor for fraction addition.

There are four main phases for authoring an Example-tracing tutor:

1. Demonstrate: in the student interface, define the initial problem state and demonstrate examples of correct and incorrect behavior for each problem for which students will be tutored. Alternate ways of solving a problem are recorded as separate paths in the graph.

2. Generalize: define ways in which the problem graph can recognize a broader range of behavior. Methods for this include specifying ordering constraints so that students can solve a problem in orders other than the one in which the problem was demonstrated; enhancing 'matching' to generalize the conditions under which a demonstrated step is matched; and marking certain steps (links) in the graph as optional.

3. Annotate: define hints for correct student actions, feedback for student errors, and success messages for correct actions. Associate knowledge components, pieces of acquired knowledge, with steps in the graph, and view a knowledge component matrix of knowledge component occurrence by problem.

4. Test: use the tutor, playing the role of student attempting to solve the problem. If possible, field the tutor with a user or number of users to get feedback early and often.

Note

When working with an Example-tracing tutor, set Tutor Type to Example-tracing Tutor.

Demonstrate

Authoring a single problem is done by demonstrating (performing) problem-solving actions. During this phase of authoring, you will commonly have the graph showing in CTAT along with a student interface with which you'll interact.

To set an initial problem state (the start state):

1. Select File > New > New Graph.
2. Set Author Mode to Set Start State.
3. In the student interface, enter values and/or manipulate the interface to represent the starting state for that problem. Note: Depending on your interface
4. Select Graph > Create Start State.

After creating a start state for the problem, you can demonstrate problem-solving steps.

To demonstrate problem-solving steps:

1. Set Author Mode to Demonstrate.
2. Interact with the student interface. Each action will be recorded as a step in the behavior graph displayed in the graph window.

To set an arbitrary state as the student start state:

1. Right-click (Windows) or Control+Click (Mac) on a state in the behavior graph that represents a point in the problem where the student should start working.
2. Select Start State. This state, as it appears in the student interface, will be the state the student sees when they start the problem.

Tip

Setting a state other than the first state in the graph to be the "start state" can be used to provide partially worked problems or to execute untutored steps that modify the student interface. (See Dynamic Interfaces for more on defining untutored steps.)

Generally, you will demonstrate a preferred, correct solution path, and revisit states in the graph to add common student misconceptions in the form of both incorrect and suboptimal actions, as well as alternative correct paths. To jump to a state in the graph, click the state node.

Note

When demonstrating steps, links are always added to the current state (the state in the graph with the bold text and outline).

When in Demonstrate mode, all steps are initially created as "Correct Action" links. To mark an action as incorrect or suboptimal, right-click the link's label and select an option from the menu Change Action Type.

The last step of a demonstration should be a done step, that is, a step where the student signals completion of the problem by pressing a Done button.

Save your graph by selecting File > Save Graph As.

Generalize

You can generalize a complete graph to account for various problem-solving strategies, as well as a varieties of student input and step order. By generalizing, you're adding flexibility to a single graph structure to allow for a greater variety of student behavior.

Generalizing a behavior graph enables you to:

• Specify a range or set of student input values to match on a given step;
• Enable the tutor to perform some steps;
• Constrain the order in which steps should be performed by the student.

Input Matching

In an Example-tracing tutor, each action that a student performs is compared by the tutor against the set of potential actions described in the graph. Specifically, the tutor compares the student's "selection" (component name), "action" (interaction type), and "input" (entered value) with those on the links of the graph. Whether this comparison is against all links in the graph or a specific subset of links depends on the graph's ordering constraints (see Ordering Constraints).

You commonly edit student-input matching to match a wider range of student input (i.e, beyond the input you demonstrated), but you can also generalize matching on component selection and action.

To edit student-input matching:

• Double-click a link's label in the graph.

Figure 1: Edit Student Input Matching dialog.

When CTAT compares a student action with that defined on a link, it can do so precisely (an exact match), or by another method (e.g., a regular expression, range, wildcard, or any value). The type of comparison is specified on each link in the graph. By default, the type of comparison is an exact match. Other types of input matching can be specified for a link:

Exact Match
The default matching type. Requires that the student's selection, action, and input match exactly with that specified on the link.

Any Match
Any input value for the specified selection and action is considered a match.

Range Match
A value within the specified numeric range is considered a match. Includes the minimum and maximum values (e.g, 0-100 would match inputs of 0, 2, 10.12, or 100).

Wildcard Match
A value that matches the wildcard pattern is considered a match. A wildcard * denotes any number of characters. Special characters such as {, \, or & should be avoided; for these, use a regular expression match. A wildcard expression can be used for selection, action, and/or input.

• Example 1: "*_text" would match "_text", "this_text", or "123abc_text".
• Example 2: "abc*xyz" would match "abcxyz", or "abc123xyz".

Regular Expression Match
A value that is entirely matched by the regular expression is considered a match. If the regular expression matches a substring but not the whole string, it is not considered a match. (Note: this behavior will change in a future release, when the matcher will return "match" when only part of the student input matches; authors can use the regex commands '^' and '$' to preserve current behavior). Uses the JavaScript regular expression engine. A regular expression can be used for selection, action, and/or input. Regular expression matching and formula matching (below) are the most powerful types of matching in CTAT.

Below are six examples that illustrate some of the regular expression conventions. For a complete reference and tutorial, see Oracle's Java Regular Expression Lesson.

• Example 1: The regular expression ^cat$ would match "cat", but not "a cat", "cat one", or "Cat"; the ^ character specifies beginning of a line, and $ specifies end of a line.
• Example 2: The regular expression animal. would match "animals" or "animale", but not "animal" or "animales"; the . character represents any single character.
• Example 3: The regular expression [THK]im would match "Tim", "Him", or "Kim", but not "Jim"; any single character from the set of characters between the square brackets can be used in the match.
• Example 4: The regular expression a{3} would match "aaa", but not "aa"; the number between the braces specifies the number of times the preceding character must occur to match.
• Example 5: The regular expression .*hello would match "hi hello", "xhello", or "hello"; the * character represents zero or more occurrences of the preceding character, and . represents any character.
• Example 6: The regular expression \d\w\W would match "0z!" or "09@", but not "b1+" or "4!!"; \d represents a digit (0-9), \w represents a word character (a-zA-Z_0-9), and \W represents a non-word character.

Formula Match
An expression that can calculate a value (or set of values) used to test against the student's input. See Generalizing an example-tracing tutor with formulas.

Student- and Tutor-Performed Actions

Most steps in a behavior graph are steps that the student should perform while using the tutor. In certain cases, you might want the tutor itself to perform actions on the interface; these are tutor-performed steps (also referred to as "tutor-performed actions" or TPAs).

There are two types of tutor-performed steps: evaluated and unevaluated.

• Use "evaluated" tutor-performed steps when you want the tutor to enter input as if the student had done so. For example, given a fraction-addition tutor that requires converting two fractions to use the least-common denominator, if the student correctly enters the correct denominator for the first conversion, the tutor could enter the correct denominator for the second conversion.
• Use "unevaluated" tutor-performed steps when you want the tutor to perform actions on the student interface. For example, you might want to hide a group of interface components initially and show them only when the student reaches a certain point in the problem.

In the Edit Student Input Matching dialog, set the "Actor" to Tutor for evaluated TPAs or Tutor (unevaluated) for unevaluated TPAs.

Figure 2: Setting the actor for a tutor-performed step.

Tutor-performed steps are covered in more detail in Dynamic Interfaces.

Ordering Constraints

In a tutorable activity, you may want to specify which problem-solving steps come before other steps. In what order (if any) should the correct steps be completed by the student?

You can specify ordering constraints in CTAT by grouping steps, and specifying whether each group of steps can be completed in any order or sequential order. In addition, you can nest groups of steps. Using these basic tools, you can construct complex step-ordering constraints in your tutor.

The most basic example of an ordering constraint is when all steps in a problem or activity make up a single group of steps and that group is either ordered or unordered. When you create a new behavior graph, all steps are considered part of this group by default.

The Group Editor window (show below on the left) is used to create groups and specify ordering constraints.

Figure 3: The Group Editor window.

To create a group of steps (links):

1. Select links from the behavior graph by holding CTRL or SHIFT and clicking each of the links (the arrows). (Alternatively, you can click and drag to draw a bounding box around the desired links.)
2. In the Group Editor, click Create Group. Enter a name for your group.

To add or remove links from a group:

1. In the Group Editor, select the group to modify.
2. Select links in the behavior graph to add or remove.
3. In the Group Editor, click Add/Remove Links.

In the Group Editor, you can also rename a group, delete a group, or choose whether it should be visualized in the behavior graph (Displayed On Graph).

Note

Deleting a group places its links in the group above it; it does not delete steps from the behavior graph.

Ordered or Unordered

A group of steps is either ordered or unordered.

Ordered: The student has to perform the entire series of steps in the order they appear in the graph, from the start state node toward the final "done" step, for the problem-solving behavior to be considered correct.

Unordered: The student can perform the steps in any order for the behavior to be considered correct.

Ordered and unordered groups are visualized in the current behavior graph as rectangles and ovals, respectively.

Figure 4: Behavior graph visualization of ordering constraints on groups of steps.

A group of steps is either "reenterable" or "not reenterable". "Reenterable" means that once the student starts working on a step in the group, the student may "leave" that group of steps to work on others, returning to it (reentering it) later. If a group is "not reenterable", the student must complete that group of steps once she starts it.

Another way of thinking about "not reenterable" is the phrase "you start, you finish": if you start working on this group of steps, you have to finish working on all if them before you can move on.

Experiment with ordering constraints in your tutor, modifying the ordered/unordered and "reenterable" settings for each group.

Try combining ordering constraints with optional steps, formulas, and/or other types of student-input matching.

Optional Steps and Minimum/Maximum Traversals

You can set a minimum and/or maximum number of a times a step should be performed by a student. CTAT refers to these events as "traversals"―that is, a successful match, by the tutor, on a student's action.

By default, the minimum and maximum traversals for a step are both set to "1": the student should perform the step once, but not more than once. You can use this feature to designate a step as optional by setting its minimum required traversals to "0".

Minimum and maximum traversals can be set on any correct or suboptimal-action step in an example-tracing tutor.

To set minimum and maximum traversals:

1. Click a link's action label in a behavior graph and select Set min/max traversals.
2. Modify the values and press Ok.

Test your tutor to see if the change has the desired effect on your tutor's behavior.

Annotate

Behavior graphs can be annotated with hints, feedback, and skills.

Hints

When a student requests a hint, the first (top-level) hint from the relevant link is displayed. A link can have multiple levels of hints, where each successive hint request by the student displays the next level of hint.

Tip

The last hint is commonly a bottom-out hint, a hint that informs the student of the correct answer so that they can continue. This approach, however, can facilitate students "gaming" of the tutoring system by clicking through to the final hint for each step of the problem.

Note

Only correct action links can have hints associated with them. If you switch the action type of a correct action link to one of the other types, you will be prompted to choose what to do with the existing hints—you can convert them to buggy (error) messages or remove them from the link.

To define hints for a correct action link:

1. Click a correct action link in the Graph window.
2. Select Edit Hint and Success Messages.
3. Enter hints into the hint message text areas. Alternatively, you can copy hints from another link or rule by selecting the link or rule name from the drop-down boxes at the bottom of the Edit Hint and Success Messages dialog box.
4. (Optional) Click Add Hint Level to define more than three levels of hints.
5. Click Done.

Figure: Edit Hint and Success Messages dialog.

Feedback

Feedback for a student action can be specified on all types of links. It is given after a student action is evaluated by the tutor.

For correct action links, feedback is a success message. For an incorrect or suboptimal action, feedback is an error ("buggy") message.

To define a success message:

1. Click a correct action link in the Graph window.
2. Select Edit Hint and Success Messages.
3. Click More Options.
4. Enter a success message in the Please Edit Success Message text area.
5. (Optional) Choose whether or not the success message should be copied to links with a) the same selection/action/input, and/or b) the same production rule.
6. Click Done.

To define a 'buggy' (error) message:

1. Click an incorrect or suboptimal action link in the Graph window.
2. Select Edit Bug Message.
3. Enter a buggy (error) message in the text area of the dialog box. Alternatively, you can copy the buggy message from another link in the graph by selecting the link's description from the drop-down box.
4. (Optional) Choose whether or not to copy the buggy message to all links with the same selection/action/input.
5. Click OK.

Skills

Each link on a behavior graph can have any number of skill labels associated with it. A skill label is a tag used to describe a piece of acquired knowledge that is associated with a step (link) in the graph. When applied consistently to steps in a problem, skill labels facilitate many types of analysis on the data generated by student use of the tutor. Such analysis based on knowledge components can be accomplished using the PSLC DataShop, a web application that facilitates learning science research through advanced reports and queries. Using the DataShop, you can easily produce learning curves and other reports from log data coded with knowledge component labels.

To collect this data for later analysis, logging must be turned on for the tutor. See Logging and Deployment for instructions on configuring and enabling logging, including logging to the PSLC DataShop.

Note

If Knowledge Component labels are not visible in the graph, click Graph > Show Skill Labels.

To associate a knowledge component (skill) with a step:

1. Single-click the default skill ("unnamed") or another existing skill that you'd like to change.
2. Select Edit Skill Name.
3. In the Edit Skill Name dialog (figure below), enter a new skill name in the drop-down box or select an existing skill name from the drop-down menu.
4. (Optional) Enter a new skill set name or select an existing one from the drop-down menu.
5. Click OK.

Figure: Edit Skill Name

To disassociate a knowledge component from a step:

1. Single-click the knowledge component that you'd like to disassociate from the step.
2. Select Delete Skill Name from Link. You'll be asked whether or not you'd like to delete the skill name.
3. Click Yes to remove the skill name from the link; to leave the skill association as is, click No.

With knowledge component labels applied, you can view a matrix of knowledge component usage by problem.

To view a knowledge component matrix:

1. Select Windows > Skill Matrix.
2. Select a directory containing problems to scan.
3. Click Open to view the skill matrix for the problems you've selected.

The Skill Matrix (pictured below) shows problem names vertically along the left-hand side of the table and skill names horizontally across the top. As skill names are often long, shortened identifiers are used (S1, S2 ...) to represent skills; these are defined below the table. The numbers in the table are counts of the number of times each skill appears in each problem.

Figure: Skill Matrix (Knowledge Component Matrix

Test

At any time during tutor authoring, you can test the behavior of your Example-tracing tutor. To do so, make sure the Tutor Type is set to Example-tracing Tutor and Author Mode is Test Tutor (see figure below).

Figure: Tutor Type and Author Mode for testing an Example-tracing tutor

In this mode, you can use the tutor, playing the role of a student attempting to solve a problem. Feedback, hints, and other messages will be enabled.

To jump to a state of the problem—partially solved, for example—click a node in the behavior graph. The student interface will update to reflect that state. You can then continue testing the tutor from that state.

To reset the problem, click on the start state of the behavior graph (the first node in the graph).

Changing the tutor's feedback and hint messages can be done while in Test Tutor mode. To demonstrate new behavior for the problem, you will need to switch to Demonstrate mode.

Back to top

Next >> Cognitive Tutors (Jess)