cmi5 Conformance Testing Working Draft - AICC/CMI-5_Spec_Current GitHub Wiki

##LMS Testing

  1. LMS must be XAPI/LRS Conformant – (ADL is developing test suite software for this)
    • LMS vendor provides endpoint & creds to LRS
    • Direct test suite to LRS communication Rest API only
    • No client side installs
    • Test for this would be independent of LMS interface
  2. LMS must able to import a cmi5 course structure
    • The LMS MUST support course structures containing more than 1000 AUs
    • Reject no-conformant course structures
    • LMS conform to cmi5 launch (Launch Test)
      • Check for Previous abandoned session (see session mgt test scenarios)
      • Launch Statement (written to LRS before actual launch)
      • Write to State API (written to LRS before actual launch)
      • Launch Mechanism
      • All launch params (LRS end-point, actor, registration, activityID, fetchURL)
      • Implement Fetch URL
      • Verify POST method
      • Verify Launch Window (ownWindow or anyWindow) is correct based on course structure
  3. Reject multiple calls to Fetch URL (LMS receives message = 1st use)
    • Verify error codes are properly returned for each error condition.
  4. LMS conform to session management:
    • LMS must not allow AU to void statements
    • LMS must either reject or void statements that conflict with the cmi5 "Statement API"
    • All JSON properties included with non-null values MUST be recorded in the LRS.

LMS Testing Scenarios:

  • Abandoned – launch AU and abnormally terminate. Verify that the LMS writes an Abandoned statement in response to the learner launching another AU for the same registration. (Test suite may need a secondary monitor to check LRS statements with higher user credentials)

  • MoveOn (NotApplicable, PassAndCompleted, Passed, Completed) – Launch AU, set status, exit, verify that Satisfied

  • MasteryScore* – Verify mastery score appears in the State Document and in Launched Statement.

  • Multi-Session Scenario (PassedAndCompleted moveOn) – AU is launched, user completes material (Completed Statement), exits, relaunches to take embedded Asssessment, issues Passed Statement, exit.

    • Test Setup
      • Course Structure: Single AU in a Block.
      • Simulated AU design with an embedded Assessment – that generates a Passed Statement
      • Simulated AU design to issue complete Statement after viewing content.

  • Multi-Session Scenario (Completed moveOn)

  • Multi-Session Scenario (Passed moveOn)

  • Launch Mode we don’t test. (LMS determines) -

  • Verify

    1. Verify that Completed is issued in one of the sessions
    2. Verify the Passed is issued in a different session
    3. Verify after both Complete & Passed Statements are issued that the moveOn behavior (for “true”) the LMS issues a “Satisfied” statement for the block and the course.

  • Block Rollup Scenario (PassedAndCompleted moveOn) – AU is launched, user completes material (Completed Statement), exits, relaunches to take embedded Asssessment, issues Passed Statement, exit.

    • Test Setup.
      • Course Structure: Single AU in nested blocks (3-Levels).
      • Simulated AU design with an embedded Assessment – that generates a Passed Statement
      • Simulated AU design to issue complete Statement after viewing content.
    • Verify
      1. Verify that Completed is issued in one of the sessions
      2. Verify the Passed is issued in a different session
      3. Verify after both Complete & Passed Statements are issued that the moveOn behavior (for “true”) the LMS issues a “Satisfied” statement for each block and the course

  • Default “Not Applicable” MoveOn – Scenario. All AUs have no moveOn criteria (or Not Applicable criteria). Verify that Satisfied statements is issued for the course (and respective blocks) on registration.

  • Multi-Session/Multi AU Scenario (PassedAndCompleted moveOn) – A single AU is launched, user completes material (Completed Statement), exits, relaunches to take embedded Asssessment, issues Passed Statement, exit. (the other AU is not launched)

    • Test Setup.
      • Course Structure: 2 AU’s in a Block.
      • Simulated AU design with an embedded Assessment – that generates a Passed Statement
      • Simulated AU design to issue complete Statement after viewing content.

    Verify 1. Verify that Completed is issued in one of the sessions 2. Verify the Passed is issued in a different session 3. Verify that after both Complete & Passed Statements are issued for the single AU that the LMS does NOT issue a “Satisfied Statement”

  1. Per Test Suite Course structure with Blocks. Block with one AU, Block with mixed AU’s (with met and un met moveOn),
  2. Test Course Structure (whole structure) with all criteria met.
  3. LMS must conform to course structure rules
    • Must write satisfied statement for Block and Course completion
    • “MoveOn” ? – Verify that Satisfy Statements are written for given Move Criteria
    • AnyWIndow / OwnWindow

##Content Testing

  1. AU conform to cmi5 launch (Launch Test)
    • Parse Launch Parameters
    • Call Fetch URL and get Auth Token
      • If HTTP Error occurs – retry
      • If error out – then close
    • Read from State API
    • Read Agent Profile
    • Write Initialized Statement
      • Use Context Template (retrieved from StateAPI)
    • Write optional statements (conforming manner)
      • cmi5 Defined
      • cmi5 Allowed
      • Use Context Template (retrieved from StateAPI)
    • Write Terminate Statement
      • Use Context Template (retrieved from StateAPI)
  2. AU conform to session management:
    • Validation of Statements
      • ContextTemplate –Verify that all Statements contains the cmi5 context template elements as provided by state api. (additions allowed, modifications not allowed)
      • PublisherID - Verify publisherID is in correct location in the content of statements issued by the AU.
    • Conditions - Test Scenarios – Events/Sequence
      • MasteryScore – (If Score is present) Verify that Passed Statements are not issued for Scores below MasteryScore threshold. Verify that Passed Statements are issued if MasteryScore threshold is met. Verify that Failed Statements are issued when Score is below MasteryScore threshold.
  3. Verify that success property of result is set True for Passed, False for Failed.
  4. Verify the completion property is not present
    • Multiple Statements During a session. Verify that AU does not issue multiple cmi5 defined verbs in a session.
    • ReturnURL - Verify AU navigates to the ReturnURL specified (in State API) on exit.
    • Initialize – Verify that AU issues Initialized Statement as the first statement of a session.
  5. Launch Mode – Review & Browse
    • AU must not send Passed, Failed, or Completed Statements.

Agent Profile Data Model - AU must make a call to Profile API for cmiLearnerPreferences Document Course Structure

  • IRIs – Verify that course structure has valid IRI’s for all ID’s in course structrure. Test Suite Design Considerations
  • LMS/LRS – to for testing content/authoring
  • Content – matching examples
  • Content – Course structure – relative and hosted content URLS
  • Content – cmi5 content simulator
    • Random Timing