Problem Summary - CMUCTAT/CTAT GitHub Wiki
Below is a sample problem summary compiled at the end of a problem. This case shows a problem with 9 required steps (see notes below):
<ProblemSummary ProblemName="problemName" CompletionStatus="incomplete"
Correct="8" UniqueCorrect="4" UniqueCorrectUnassisted="2"
Hints="5" UniqueHints="4" HintsOnly="2"
Errors="5" UniqueErrors="4" ErrorsOnly="2"
UniqueSteps="8" RequiredSteps="9" TimeElapsed="31343">
<Skills> see more detail below... </Skills>
</ProblemSummary>| note | name | sample | description |
|---|---|---|---|
| ProblemName | aProblem | brd and start state name | |
| CompletionStatus | incomplete | "complete" means a successful Done step; else "incomplete" | |
| 1 | Correct | 8 | raw count of correct responses |
| 2 | UniqueCorrect | 4 | unique steps with only hint or correct responses |
| 3 | UniqueCorrectUnassisted | 2 | unique steps with only correct responses, no hints |
| 4 | Hints | 5 | raw count of hint requests, not counting 2nd- or other-level hints |
| 5 | UniqueHints | 4 | unique steps with hint requests |
| 6 | HintsOnly | 2 | unique steps with hint requests but no errors |
| 7 | Errors | 5 | raw count of incorrect responses |
| 8 | UniqueErrors | 4 | unique steps with error responses |
| 9 | ErrorsOnly | 2 | unique steps with error responses and no hint requests |
| 10 | UniqueSteps | 8 | number of different steps attempted |
| 11 | RequiredSteps | 9 | number of steps required on the best path towards the first Done step |
| 12 | TimeElapsed | 31343 | milliseconds from problem initialization until this report |
Notes:
- The student had to eventually answer all 8 steps correctly to proceed.
- On 4 different steps, the student may have requested a hint, but committed no error.
- On 2 different steps, the student answered correctly on the first try without requesting a hint.
- The student asked for a top-level hint 5 times, not counting requests for 2nd-level hints.
- The 5 hint requests were for 4 different steps (student requested a hint on 1 step twice).
- On 2 different steps, the student did request a hint, but committed no error.
- The student entered incorrect answers 5 times.
- The student entered incorrect answers on 4 different steps (got 1 step wrong twice).
- On 2 different steps, the student entered an incorrect answer but didn't request a hint.
- The student has attempted 8 steps so far.
- There are 9 required student steps (omitting tutor-performed or steps with minimum traversals = 0) on the preferred path from the start state to a Done step.
- The elapsed time from the student downloading this problem during this session until the time of this report is 31.343 seconds.
In addition to the summary information, the tutor transmits updated values for the skills of interest. The attributes for each skill are as follows:
| Name | Description |
|---|---|
| name | Unique name for the skill within the category (namespace) |
| label | Name visible to the student in the skill meter display |
| category | Namespace for the skill name |
| description | Description shown on skill reports |
| pKnown | The probability that the student has mastered the skill |
| pGuess | The probability that the student would guess correctly without having mastered the skill |
| pSlip | The probability that the student would err after having mastered the skill |
| pLearn | The probability that the student would come to master the skill |
| history | A bit vector showing the student's most recent performance on steps demonstrating this skill |
More explanation on the history attribute:
The new history attribute is a bit vector recording recent performance-based updates to its associated skill. Each time the skill is updated, the existing history value is shifted left 1 bit and the new performance result (1 for correct, 0 for hint or incorrect) is stored in the history's least significant bit. Note that history is updated only when pKnown is updated--that is, only on the 1st attempt at a step. The TutorShop maintains the history value with the other skill attributes in the student model; it provides all current attribute values at problem start time. During problem execution, the tutoring service updates the value and, at the end of a problem, sends it in the problem summary's skills element to the TutorShop.
For example, if skill S is associated with 2 links having SAIs (tf1,UpdateTextField,one), (tf2,UpdateTextField,two) and initially has history 3 (11 in binary), you could get this sequence:
| Selection | Action | Input | Grade | History | Notes |
|---|---|---|---|---|---|
| tf1 | UpdateTextField | hint | 6 | binary 110: shift left 1 bit and insert 0 (for hint or incorrect) as lsb | |
| tf1 | UpdateTextField | wrong | incorrect | 6 | no change: not 1st attempt at step |
| tf1 | UpdateTextField | one | correct | 6 | no change: not 1st attempt at step |
| tf2 | UpdateTextField | two | correct | 13 | binary 1101: shift left 1 bit and insert 1 (for correct) as lsb |
Here are example skills from an exercise in adding fractions:
<Skills>
<Skill name="reduce-numerator" label="reduce-numerator" category="fraction-addition"
description="Calculate numerator when reducing a fraction to lowest terms."
pKnown="0.25" pGuess="0.1" pSlip="0.2" pLearn="0.15" />
<Skill name="reduce-denominator" label="reduce-denominator" category="fraction-addition"
description="Calculate denominator when reducing a fraction to lowest terms."
pKnown="0.25" pGuess="0.1" pSlip="0.2" pLearn="0.15" />
<Skill name="copy-answer-denominator" label="copy-answer-denominator" category="fraction-addition"
description="Copy the common denominator from the addends to the sum."
pKnown="0.25" pGuess="0.1" pSlip="0.2" pLearn="0.15" />
<Skill name="add-numerators" label="add-numerators" category="fraction-addition"
description="Add numerators after addends have a common denominator."
pKnown="0.25" pGuess="0.1" pSlip="0.2" pLearn="0.15" />
<Skill name="convert-numerator" label="convert-numerator" category="fraction-addition"
description="Calculate numerator when converting addends to have a common denominator."
pKnown="0.25" pGuess="0.1" pSlip="0.2" pLearn="0.15" />
<Skill name="multiply-denominators" label="multiply-denominators" category="fraction-addition"
description="Multiply the denominators to calculate a common denominator."
pKnown="0.25" pGuess="0.1" pSlip="0.2" pLearn="0.15" />
<Skill name="determine-lcd" label="determine-lcd" category="fraction-addition"
description="Calculate the least common denominator."
pKnown="0.25" pGuess="0.1" pSlip="0.2" pLearn="0.15" />
</Skills>How some CTAT data elements could be used in SCORM:
| SCORM Data Model | CTAT Data Element | Notes |
|---|---|---|
| cmi.score | correct count | The preferred score in SCORM is cmi.score.scaled, which is a number in the interval [-1.0,1.0]. In CTAT this could be calculated various ways, including uniqueCorrectUnassisted/uniqueStepCount. SCORM also permits a raw score, which can be any numeric value. |
| cmi.completion_status | CompletionStatus | Both SCORM and CTAT have statuses of not attempted, incomplete and complete. |
| cmi.interactions | steps | Each interaction could be a step attempt (a single {selection,action,input}) or a step (identified by {selection,action}). |
| cmi.objectives | knowledge components (skills) | Both SCORM objectives and CTAT knowledge components refer to the acquisition of competence in some domain-specific subfield. Both use a numeric metric between 0.0 and 1.0, both have a concept of mastery, both can be used for problem sequencing. |
