Skill Attributes and Behavior - CMUCTAT/CTAT GitHub Wiki

A page on what attributes CTAT skills have, how they should behave, and where they're recorded.

Skill Attributes

The attributes for each skill are as follows. The current pKnown value is updated step-by-step (see also notes on behavior, below) and shown on the CTAT Skill Meter.

Name	Description
name	Unique name for the skill within the category (namespace)
label	Name visible to the student in the skill meter display
category	Namespace for the skill name
description	Description shown on skill reports
pKnown	The probability that the student has mastered the skill
pGuess	The probability that the student would guess correctly without having mastered the skill
pSlip	The probability that the student would err after having mastered the skill
pLearn	The probability that the student would come to master the skill
history	A bit vector showing the student's most recent performance on steps demonstrating this skill

More explanation on the history attribute:

The history attribute is a bit vector recording recent performance-based updates to its associated skill. Each time the skill is updated, the existing history value is shifted left 1 bit and the new performance result (1 for correct, 0 for hint or incorrect) is stored in the history's least significant bit. Note that history is updated only when pKnown is updated--that is, only on the 1st attempt at a step. The TutorShop maintains the history value with the other skill attributes in the student model; it provides all current attribute values at problem start time. During problem execution, the tutoring service updates the value and, at the end of a problem, sends it in the problem summary's skills element to the TutorShop.

For example, if skill S is associated with 2 links having SAIs (tf1,UpdateTextField,one), (tf2,UpdateTextField,two) and initially has history 3 (11 in binary), you could get this sequence:

Selection	Action	Input	Grade	History	Notes
tf1	UpdateTextField		hint	6	binary 110: shift left 1 bit and insert 0 (for hint or incorrect) as lsb
tf1	UpdateTextField	wrong	incorrect	6	no change: not 1st attempt at step
tf1	UpdateTextField	one	correct	6	no change: not 1st attempt at step
tf2	UpdateTextField	two	correct	13	binary 1101: shift left 1 bit and insert 1 (for correct) as lsb

Here are example skills from an exercise in adding fractions:

</Skills>

Notes on Skill Behavior

Whenever the UI issues a ProblemSummaryRequest, CTAT should include the current p-knowns for the skills of interest, as revised by the student's performance on this problem, in the ProblemSummaryResponse.
Skill p-known maintenance.
1. A skill should change only on the 1st attempt or hint request on a step. The skill code should maintain a list of selection-action pairs and increment or decrement a skill only if it has not altered that skill on that selection-action before.
2. The relevant selection-action on an error is that from the silent hint cycle, not from the student.
3. The relevant selection-action on a hint request is that finally returned in the hint response, even if the student asked for a hint on a different selection.
4. If a skill is attached to 2 or more links (with different selection-actions) then it should change when the student 1st tries a 2nd or subsequent step with that skill.
5. Generally, skills decrement on an error or hint request and increment on a correct response. But if a p-known gets low enough, you can get an increment even on an error.
6. Mastery, by default, is p-known >= .95. When you reach mastery, the skill bar should change to green.
Notes on what skills to maintain.
1. A skill name is in 2 parts name category, where each of name and category can’t have any embedded spaces. There can be several names for a single category. It’s possible to omit category. If name is “unnamed” and category is empty, then this is just a placeholder for the CTAT Author UI; it’s not a real skill.
2. When an author attaches skills to a link in a graph, CTAT records the association in <rule> child elements inside the <edge> elements.
3. When the brd is saved, CTAT creates an inventory of skills and records it in <productionRule> elements near the bottom of the brd. The elements include an opportunities attribute showing how many links a skill is attached to. TutorShop uses this inventory to characterize this problem for its skill-based adaptive problem selection.
4. When the author creates a problem set in TutorShop, he or she chooses the skills of interest for the entire set. By default, these are the union of all the skills in the problem set’s brd <productionRule> inventories. But often that’s too long a list (you generally don’t want a skill meter with 47 skill bars, e.g.). So the author selects a subset of, say, 2-8 skills of interest for this problem set.
5. When TutorShop sends CTAT a problem, the FlashVars (forwarded to CTAT in the SetPreferences msg) contain the skills of interest. Only these appear in the skill meter and only these need maintenance step-by-step.
6. The changes to p-knowns step-by-step are transmitted to the student UI in the skillBarVector property of the AssociatedRules msg. Each element of the vectory has the following form [the backquote (`) delimiter is specified in a separate AssociatedRules property]:
  name category`pknown`mastery
  where--
  name category are the skill name and category;
  pknown is a decimal in the range [0..1];
  mastery is 0 if unmastered, 1 if mastered (means skill bar should turn green)

Skill Attributes and Behavior - CMUCTAT/CTAT GitHub Wiki

Skill Attributes

Notes on Skill Behavior

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️