TutorShop Interface to Custom Problem Selectors - CMUCTAT/CTAT GitHub Wiki
The TutorShop Learning Management System, which was built to deploy CTAT tutors on the web, groups individual tutor instances, or "problems" (a problem is a pairing of a student interface and a behavior graph), into problem sets at runtime. Each problem set has a selection algorithm that determines which of its problems will be shown to the student and in what sequence. The TutorShop itself supports several selection algorithms, including fixed sequences, random selection from within the problem set, and adaptive selection by skills.
The TutorShop also provides an interface for authors to use their own problem selection algorithms. Custom problem selectors are implemented as separate programs running on a server, using any language the author desires. Before presenting any problem to a student, the TutorShop sends a request to the custom problem selector and expects in reply the name of next the problem to present from the current problem set. This exchange occurs once before the student sees the first problem of a session and once after each problem has been completed.
Integrating custom problem selectors
Custom problem selectors come in two types, based on the technology used to interface them with TutorShop:
-
External selectors: these are programs that run on the same server as TutorShop and are called through a command line string. Parameters are sent from TutorShop using the standard input channel, results are sent back using the standard output channel.
-
Remote selectors: there are programs that can run either on the TutorShop server or on a separate server, and are called through an HTTP url. Parameters are send in the request body, results are sent back in the response body.
Remote servers are more difficult to implement, as they need to include an HTTP server interface, but have the advantage that they can run from any server.
In order to use custom selectors, the author needs to go to the problem set Settings page in TutorShop and set the Selection Algorithm to either External Selection or Remote Selection, and then set the Selection Command to either the command string or the launch url, respectively.
Messages exchanged with custom problem selectors
For both types of custom problem selectors the requests to the external program and the expected responses are JSON objects with the following top level keys:
Request:
- skills: a list of objects representing current states of knowledge components associated with the student;
- detectors: a list of objects representing detector variables associated with the student;
- dynamic_model: an object, initially empty, representing a student model extension, which can be used by the custom selector to store key/value pairs between runs;
- problem_set: an object representing the current problem set that a problem selection is requested for;
- problems: a list of problem objects representing selection candidates.
Response:
- problem_name: a string specifying the selected problem name;
- dynamic_model: an object representing the updated student model extension.
A sample request and response are shown below. First, the JSON request from TutorShop to the custom problem selector:
{
"skills":[
{
"skill_id":414,"p_initial":1.0,"p_known":1.0,"initial_opportunity_count":3,"opportunity_count":12,"history":3555,
"user":{
"school_id":14,"standard_id":101,"user_role_id":8,"first_name":"student","last_name":"0","middle_name":"",
"username":"stu0","email_address":"","date_of_birth":"1998-08-23","status":"Active"
}
},{
"skill_id":412,"p_initial":0.93,"p_known":0.93,"initial_opportunity_count":1,"opportunity_count":4,"history":13,
"user":{
"school_id":14,"standard_id":101,"user_role_id":8,"first_name":"student","last_name":"0","middle_name":"",
"username":"stu0","email_address":"","date_of_birth":"1998-08-23","status":"Active"
}
}
],
"detectors":[],
"dynamic_model":{"+=":0.01,"for":0.02},
"problem_set": {
"package_id":264,"name":"For Loop Test3","label":"For Loop Test3","description":"For Loop Test3","picture":null,
"selection_algorithm":"remote_selection","selection_command":"http://pawscomp2.sis.pitt.edu/bn/GetNextProblem",
"max_count":1000,"max_repeat":1,"initial_sequence":0,"team_size":1,"status":"Active"
},
"problems":[
{
"package_id":264,"name":"foraddasm_s1","label":"foraddasm_s1","description":"For with Add Assignment",
"problem_file":"foraddasm_itgt_s1.brd","student_interface":"foraddasm_s1.html","student_interface_class":null,
"interface_type":"html","tutor_flag":"tutor","selection_features":"","completed_count":0
},{
"package_id":264,"name":"foraddasm_s2","label":"foraddasm_s2","description":"For with Add Assignment",
"problem_file":"foraddasm_itgt_s2.brd","student_interface":"foraddasm_s2.html","student_interface_class":null,
"interface_type":"html","tutor_flag":"tutor","selection_features":"","completed_count":0
},{
"package_id":264,"name":"foraddasm_s3","label":"foraddasm_s3","description":"foraddasm_s3",
"problem_file":"foraddasm_itgt_s3.brd","student_interface":"foraddasm_s3.html","student_interface_class":null,
"interface_type":"html","tutor_flag":"tutor","selection_features":"","completed_count":0
}
],
"condition":""
}
And second the expected response from the custom problem selector to the TutorShop. It includes (a) what problem to show the student next and (b) what dynamic model data to save and return in the next request to the custom problem selector:
{
"problem_name":"foraddasm_s2",
"dynamic_model":{
"+=":0.02,"for":0.03
}
}
Request and response details
Skills array
A skill, also called a knowledge component, represents the student's mastery of a concept or procedure in the problem. Skills are typically assigned to individual problem steps; a step can be associated with any number of different skills, and a skill can be associated with any number of steps, across problems, problem sets and packages. The mastery is expressed as a probability.
The array includes an element for each skill whose value might have been recalculated in the most recent problem set. Each array element includes a skill identifier and a number p_known, the most-recently-calculated probability that the student has mastered the skill. The initial_probability property is the value of that probability at the beginning of the problem set.
The history property's value is an integer which encodes a bit vector recording the student's performance on that skill. The bit vector is revised only on a student's first attempt on a new step associated with that skill; second and later attempts on the step will not change the bit vector. A 1 bit records a correct answer, a 0 bit records a hint request or incorrect answer. The latest result is in the vector's least significant bit (LSB). Each time the tutor revises the history, it shifts prior results leftward by 1 bit position. The opportunity_count property is incremented once each time a new bit is appended to the vector: hence the difference between the count at the end of a problem and the count at the end of the prior problem is the number of updates in the most recent problem. (N.B.: the initial_opportunity_count property is different; it changes only at the start of a problem set, not on each problem).
For example, using the 2nd skill in the sample request above,
{ "skill_id":412,"p_initial":0.93,"p_known":0.93,"initial_opportunity_count":1,"opportunity_count":4,"history":13, ...}
the history value, decimal 13, is 1101 in binary, meaning that among the student's last 4 opportunities at any step to learn or demonstrate skill 412, the first attempts at that step were scored as follows (right-to-left--that is, latest listed first):
| bit position | opportunity at step using this skill | bit value | score |
|---|---|---|---|
| LSB | most recent opportunity | 1 | correct on 1st attempt at step |
| LSB << 1 | next most recent opportunity | 0 | incorrect or hint on 1st attempt |
| LSB << 2 | 2 opportunities ago | 1 | correct on 1st attempt |
| LSB << 3 | 3 opportunities ago | 1 | correct on 1st attempt |