<article class="asq-rubric" data-asq-self-assessment data-asq-peer-assessment>
  <h3>Section title</h3>
  <ol>
    <li data-asq-rubric-points="5" data-asq-rubric-label="Excellent">
       Description for 5 points
    </li>
    <li data-asq-rubric-points="4" data-asq-rubric-label="Good">
       Description for 4 points
    </li>
    <li data-asq-rubric-points="3" data-asq-rubric-label="Average">
       Description for 3 points
    </li>
    <li data-asq-rubric-points="2" data-asq-rubric-label="Sufficient">
       Description for 2 points
    </li>
    <li data-asq-rubric-points="1" data-asq-rubric-label="Insufficient">
       Description for 1 points
    </li>
  </ol>
  <!-- ... -->
</article>

An element with the class asq-rubric must be defined inside an element with assessment class. If there are many elements with the class asq-rubric, only the first one found will be used.

A rubric can be used for self-assessment, peer-assessment, both or none. using the corresponding data-asq-self-assessment and data-asq-peer-assessment attributes.

Note If none of the above attribute are given, the rubrics will still be processed by the ASQ server but it will not be used during sessions.

A rubric can have many sections. Each section is defined with an h3 title and an organized list ol of criteria. Each criterion can have a specific amount of points attributed with the data-asq-rubric-points attribute. The number of points can be hidden from the grader by defining data-asq-rubric-label which replace the number of points with the value of the label.

Note If the data-asq-rubric-points attribute is omitted, the criterion is given as points the size of the list of criterion minus its position in the list. This means, for a list of four elements, the first one has 4 points, the second one has 3 points, the third one has 2 points and the fourth one has 1 point.

Note Even with a label, a criterion still has a point value which is taken into account in the overall grade.

<article class="asq-rubric" data-asq-self-assessment data-asq-peer-assessment>
  <h3>Argument</h3>
  <ol>
    <li data-asq-rubric-points="5" data-asq-rubric-label="Excellent">
        Argument pertains to relationship between social factors and educational opportunity and is clearly stated and defensible.
    </li>
    <li data-asq-rubric-points="4" data-asq-rubric-label="Good">
      Argument pertains to relationship between social factors and educational opportunity and is defensible, but it is not clearly stated.
    </li>
    <li data-asq-rubric-points="3" data-asq-rubric-label="Average">
      Argument pertains to relationship between social factors and educational opportunity but is not defensible using the evidence available.
    </li>
    <li data-asq-rubric-points="2" data-asq-rubric-label="Sufficient">
      Argument pertains to relationship between social factors and educational opportunity and is defensible, but it is not clearly stated.
    </li>
    <li data-asq-rubric-points="1" data-asq-rubric-label="Insufficient">
      Social factors and educational opportunity are discussed, but no argument is presented.
    </li>
  </ol>
</article>

The assessment process for a question only happens if a rubric is defined for said question and if at least one of the self- or peer-assessment data attribute is given.

The assessment process happens immediately after the viewer submits his answer. First, if enabled, the viewer will have to assess his own answer. Then, once he sends his answer, he will receive the answer of another viewer anonymously. The algorithm for this is independent of the rubric logic. Once the viewer has finished the assessment of that answer and sends it, he will receive the next one. This process goes as long as there are answers to be assessed or until the presenter stops the process.

The screen will be split in two vertically. On the left side will be the answer which needs to be assessed. On the right will be the rubric and at the bottom the current grade given by the viewer.

TODO: put a doodle here

Note A rubric document in the database corresponds to a rubric section in the definition above. A rubric is split into sections and each section is saved independently.

A rubric is composed of the reference to its question (qid) as an object id, the title of the rubric as a string and an array of criterion which is composed of sub documents following the criterionSchema schema. The schema for a singe criterion is composed of a label as a string, a point which represent the point value of the criterion as a positive number and a description of the criterion as a string.

Note If the label is not provided it is set to the string representation of the point field using a pre save middleware.

var rubricSchema = new Schema({
  qid       : { type: OjbectId, ref:'Question', required: true }
  title     : { type: String, required: true },
  criteria : { type:[criterionSchema], required: true } 
});

var criterionSchema = new Schema({
  label : { type: String, require: true },
  point : { type:Number, min: 0, required: true },
  desc  : { type: String, required: true }
});

// Set the default value of the label to the value of the points.
criterionSchema.pre('save' function onSave(next) {
  if (! this.label) {
    this.label = this.point.toString();
  }
  next();
});

The assessment of an answer is stored using the assessmentSchema. It is composed of the session, answer, assessee and assessor fields which refer to respectively the session, the answer, the assessee and the assessor of the assessment.

The assessment also stores: the score given to the answer by the assessor as a positive number, the type of assessment (self-, peer- or pro-assessment) and details of the assessment using the assessmentDetailSchema.

The assessmentDetailSchema stores the label and description of a part of the assessment as strings as well as the score given for that specific part of the assessment.

Note The assessee and assessor refer to the whitelist entry of the of the users and not the users directly. This allows to keep a trace if they are temporary users which are deleted after some time

Note For a given answer by a specific user, there can be at most one assessment of that answer by another specific user. This is enforced by a compound index using the assessor, the assessee and the answer.

var assessmentDetailSchema = new Schema({
  label : { type: String, require: true },
  score : { type:Number, min: 0, required: true },
  desc  : { type: String, required: true }
});

var assessmentSchema = new Schema({
  session  : { type: ObjectId, ref: 'Answer', required: true },
  answer   : { type: ObjectId, ref: 'Answer', required: true },
  assessee : { type: String, required: true },
  assessor : { type: String, required: true },
  score    : { type: Number, min: 0, max: 5, required: true },
  type     : { type: String, lowercase: true, enum: [ 'self', 'peer', 'pro' ],
               required: true },
  details  : { type: [assessmentDetailSchema], default: [] }
});

assessmentSchema.index({
  answer   : 1,
  assessee : 1,
  assessor : 1,
}, { unique : true });

• Grading Rubrics:Examples of Rubric Creation - University of Berkeley.
• Scott Klemmer: Peer and Self Assessment: How and Why (Youtube video) - HCI Group Stanford.
• Kulkarni et al. Peer and Self Assessment in Massive Online Classes (PDF) - HCI Group Stanford.
• Rubric example - HCI Group Stanford.