CMI 5 Working Group Meeting Minutes – September 30th, 2016 - AICC/CMI-5_Spec_Current GitHub Wiki

cmi5 Working Group Meeting Minutes – September 30th, 2016

Attendees

  • Bill McDonald - cmi5 working group leader
  • Benard Bouyt - Airbus
  • Andy Johnson - ADL
  • Art Werkenthin - RISC
  • Ben Clark - Rustici Software
  • Brian Miller - Rustici Software
  • Henry Ryng - inXsol
  • Christopher Thompson - Medcom Inc
  • David Pesce - Exputo
  • Dennis Hall - Learning Templates
  • Ray Lowery - Pratt & Whitney
  • Charles Touron - ATSC
  • Billy White - Thrivist

Notes

This week the group discussed common cmi5 implementation mistakes that content authoring tools.

The group also continued to discuss cmi5 conformance testing.

Common cmi5 conformance mistakes

The early adopters of cmi5 in the group shared common cmi5 (conformance) mistakes that they are currently seeing from content tool developers. They are as follows:

  1. Disregarding the returnURL - The content (AU) does not redirect the browser to the returnURL when exiting. This is a violation of the minimum compliance requirements of cmi5. The consequence of this mistake is rather significant, as the learner often cannot return to the the LMS user interface after exit.

  2. Using the Activity ID as Publisher ID - The content (AU) does not use the publisher ID provided in the State API in context activities of the statements it makes, incorrectly using the LMS generated Activity ID instead. This is a violation of the minimum compliance requirements of cmi5. The consequence of this mistake is that the data in statements are incorrectly documented.

  3. Disregarding the masteryScore - The content (AU) does not issue the proper statements (Passed) when the score is above the masteryScore (as provided in the State API by the LMS). (Instead using its own internally specified score threshold to determine mastery.) This is a violation of the minimum compliance requirements of cmi5. The consequence of this mistake is that the content is incorrectly issuing failed statements when the learner has actually achieved mastery as specified by the course definition.

In the absence of a cmi5 test suite and conformance testing services, the cmi5 developers in the group agreed to continue their efforts educating the rest of the developer community on correct cmi5 implementation.


cmi5 conformance Testing


LMS Testing

  1. LMS must be XAPI/LRS Conformant – (ADL is developing test suite software for this)

    • LMS vendor provides endpoint & creds to LRS
    • Direct test suite to LRS communication Rest API only
    • No client side installs
    • Test for this would be independent of LMS interface
  2. LMS must able to import a cmi5 course structure

    • The LMS MUST support course structures containing more than 1000 AUs
    • Reject no-conformant course structures
    • LMS conform to cmi5 launch (Launch Test)
      • Check for Previous abandoned session (see session mgt test scenarios)
      • Launch Statement (written to LRS before actual launch)
      • Write to State API (written to LRS before actual launch)
      • Launch Mechanism
      • All launch params (LRS end-point, actor, registration, activityID, fetchURL)
      • Implement Fetch URL
      • Verify POST method
      • Verify Launch Window (ownWindow or anyWindow) is correct based on course structure
  3. Reject multiple calls to Fetch URL (LMS receives message = 1st use)

    • Verify error codes are properly returned for each error condition.
  4. LMS conform to session management:

    • LMS must not allow AU to void statements
    • LMS must either reject or void statements that conflict with the cmi5 "Statement API"
    • All JSON properties included with non-null values MUST be recorded in the LRS.
    • Test Scenarios:
      • Abandoned – launch AU and abnormally terminate. Verify that the LMS writes an Abandoned statement in response to the learner launching another AU for the same registration. (Test suite may need a secondary monitor to check LRS statements with higher user credentials)
      • MoveOn (NotApplicable, PassAndCompleted, Passed, Completed) – Launch AU, set status, exit, verify that Satisfied
      • MasteryScore – Verify mastery score appears in the State Document and in Launched Statement.
  5. Per Test Suite Course structure with Blocks. Block with one AU, Block with mixed AU’s (with met and un met moveOn),

  6. Test Course Structure (whole structure) with all criteria met.

  7. LMS must conform to course structure rules

    • Must write satisfied statement for Block and Course completion
    • “MoveOn” ? – Verify that Satisfy Statements are written for given Move Criteria
    • AnyWIndow / OwnWindow

Content Testing

  1. AU conform to cmi5 launch (Launch Test)

    • Parse Launch Parameters
    • Call Fetch URL and get Auth Token * If HTTP Error occurs – retry * If error out – then close
    • Read from State API
    • Read Agent Profile
    • Write Initialized Statement * Use Context Template (retrieved from StateAPI)
    • Write optional statements (conforming manner) * cmi5 Defined * cmi5 Allowed * Use Context Template (retrieved from StateAPI)
    • Write Terminate Statement * Use Context Template (retrieved from StateAPI)
  2. AU conform to session management:

    • Validation of Statements * ContextTemplate –Verify that all Statements contains the cmi5 context template elements as provided by state api. (additions allowed, modifications not allowed) * PublisherID - Verify publisherID is in correct location in the content of statements issued by the AU.
    • Conditions - Test Scenarios – Events/Sequence * MasteryScore – Verify that Passed Statements are not issued for Scores below MasteryScore threshold. Verify that Passed Statements are issued if MasteryScore threshold is met. Verify that Failed Statements are issued when Score is below MasteryScore threshold.
  3. Verify that success property of result is set True for Passed, False for Failed.

  4. Verify the completion property is not present

    • Multiple Statements During a session. Verify that AU does not issue multiple cmi5 defined verbs in a session.
    • ReturnURL - Verify AU navigates to the ReturnURL specified (in State API) on exit.
    • Initialize – Verify that AU issues Initialized Statement as the first statement of a session.

Course Structure

 * IRIs – Verify that course structure has valid IRI’s for all ID’s in course structrure.

Test Suite Design Considerations

 * LMS/LRS – to for testing content/authoring
 * Content – matching examples
 * Content – Course structure – relative and hosted content URLS
 * Content – cmi5 content simulator
       * Random Timing

Best Practices

The following are new cmi5 Best Practices proposed as a result of compliance testing discussions:

 * The Fetch URL Should be unique for each session
 * Don’t recycle Auth tokens returned from Fetch URL
 * moveOn criteria NotApplicable should not be used in a course structure unless the AU is strictly optional. (a known problem with the default moveOn value.)
 * Always specify a MoveOn Criterion in the course structure for each AU.

All Previous CMI-5 Meeting Minutes

https://github.com/AICC/CMI-5_Spec_Current/wiki

CMI-5 on GitHub:

https://github.com/AICC/CMI-5_Spec_Current

⚠️ **GitHub.com Fallback** ⚠️