MeetingNotes Feb 02 2018 - AICC/CMI-5_Spec_Current GitHub Wiki

_Disclaimers/Limits_

Because compliance cannot be verified – it does not mean that an LMS (or AU) is not cmi5 conformant.

Failure to import course structures ( with over 1000 AU's ) due to size/capacity of the system does not indicate failure of compliance with cmi5. (

Test suite AU failure to execute due to software/environment conflicts…

LMS

  • (TestSuite) Create Test Course Structure(s)
    • Multiple course structures
      • ZIP Package vs XML only
      • Mix of Relative and Fully Qualified URLS

      • One for size test 1000 + AU's

      • Smaller one for Completion (Satisfied Test)
        • Large enough to test all MoveOn Criteria
          • Blocks, AU's
  • (Human) Import that structure(s) into the LMS (admin user)
  • (Human) Register the learner (admin User)
  • (Human) Access the course as a learner
    • Selected AU's (not all)
  • (Machine) AU's (Testsuite) inspect LMS/LRS data
  • (Human) Procedure for learner to follow thru the course(s).
    • Initial entry
    • Re-entry of AU's
  • (Human) Modify course structure (admin user) and verify effect as a learner
    • Eg, MasteryScore, MoveOn, etc

AU (Content) - Individually

  • (Human) add a reference (URL) to AU to inspect
  • (Human) Launch the AU and Interact with AU
  • (Human) Re-entry to AU's
    • Launch in different modes
      • Review, Normal, Browse
  • (Machine) Testsuite will inspect AU xapi data

AU (Content) – Course Structure (on or more)

  • (Machine) Testsuite (LMS) – import/validate a course structure
  • (Human) Launch each AU in the course and Interact with AU
  • (Human) Re-entry to AU's
    • Launch in different modes
      • Review, Normal, Browse
  • (Machine) Testsuite will inspect AU xapi data

Course Structure/Package

  • (Machine) Inspect XML
    • XSD validation
    • Relative pathing & Fully Qualified URL
  • (Machine) Inspect Package
    • Inspect ZIP
    • File existence validation (ZIP)
      • Content
      • XML file
    • ZIP format (32/64)

Testing Procedures

Small course testing

  •  (Human) Import that structure(s) into the LMS (admin user)
  •  (Human) Register multiple learners (admin User) – User 1, 2, 3….
  •  (Human) Access the course as a learner 1
    • oo Launch AU's in order
      •  Do Not Launch (AU5 – Not applicable)
    • oo Complete all AU's with minimum MoveOn Criterial
    • oo Single Entry
    • oo Final Check AU (Assess results)
      •  Run AU x after course complete (check for all satisfied)
      •  A wait period should be established
      •  AU x is Not Applicable
      •  AU x is a test suite specific tool (not Normal AU scenario)
      •  Alternate Credentials to Access to LRS for LMS issued statements.
        •  AUx may not have sufficient access to query the LRS for statements issued by other AU's – So alternate credentials would need to be provided (e.g. AUx will need to have a login form for alternate access- for "deep queries")
  •  (Human) Access the course as a learner 2
    • oo Launch AU's in order
    • oo Complete all AU's with MoveOn Criteria
    • oo Multiple Entry
    • oo Final Check AU (Assess results
      •  Run AU x after course complete (check for all satisfied)
      •  A wait period should be established
      •  AU x is Not Applicable
      •  Alternate Credentials to Access to LRS for LMS issued statements.
  •  (Human) Access the course as a learner 3
    • oo Launch AU's in any order
    • oo Multiple Entry (Re-launching - Exiting and Re-entering AU)
    • oo Complete all AU's except AU6
      •  Partially Complete AU6 and re-enter
      •  Complete AU6
    • oo Final Check AU(Assess results
      •  Run AU x after course complete (check for all satisfied)
      •  A wait period should be established
      •  AU x is Not Applicable
      •  Alternate Credentials to Access to LRS for LMS issued statements.

Large course testing

  •  (Machine) AU's (Testsuite) inspect LMS/LRS data
  •  (Human) Procedure for learner to follow thru the course(s).
    • oo Initial entry
    • oo Re-entry of AU's
  •  (Human) Modify course structure (admin user) and verify effect as a learner
    • oo Eg, MasteryScore, MoveOn, etc
  •  Final Check AU(Assess results)

<?xml version="1.0" encoding="utf-8"?>

<courseStructure xmlns="https://w3id.org/xapi/profiles/cmi5/v1/CourseStructure.xsd">

<course id="http://course-repository.example.edu/identifiers/courses/02baafcf">

&lt;title&gt;

  &lt;langstring lang=&quot;en-US&quot;&gt;Intro to Web Based Content&lt;/langstring&gt;

&lt;/title&gt;

&lt;description&gt;

  &lt;langstring lang=&quot;en-US&quot;&gt;

    This course will introduce you into the basics integrating xAPI with web

    based content.

  &lt;/langstring&gt;

&lt;/description&gt;

</course>

<block id=" http://www.example.com/identifiers/aublock/005430bf-b3ba-45e6-b47b-d629603d83d8" >

<au id= http://course-repository.example.edu/identifiers/courses/02baafcf/aus/4c07

 launchMethod=&quot;OwnWindow&quot;

 masteryScore=&quot;0.8501&quot;

 moveOn=&quot;Passed&quot;

 activityType=&quot;http://adlnet.gov/expapi/activities/media&quot;

>

&lt;title&gt;

  &lt;langstring lang=&quot;en-US&quot;&gt;Guess a Number&lt;/langstring&gt;

&lt;/title&gt;

&lt;description&gt;

  &lt;langstring lang=&quot;en-US&quot;&gt;

    This game uses xAPI to track the start and stop as well as the player&#39;s

    guesses.

  &lt;/langstring&gt;

&lt;/description&gt;

&lt;url&gt;./game.html&lt;/url&gt;

<launchParameters>xyz123</launchParameters>

<entitlementKey>xyz-123-9999</entitlementKey>

</au>

</block>

</courseStructure>

  • Rollup discussion – Blocks with Blocks only as children

Small Course Structure test (LMS)

  • Created by testsuite (Static or generated)
  • Zip File with content
  • Mix of relative and fully qualified URL's
  • Nested Blocks
  • Mixed Blocks - Blocks with AU's and Block Children
  • Blocks with Block children only

Block1

AU0

  • launchMethod="OwnWindow"
  • moveOn=" Completed "
  • <Fully Qualified url>
  • <launchParameters>
  • <entitlementKey>

Block1.1

AU1

  • launchMethod="OwnWindow"
  • masteryScore="0.8501"
  • moveOn="Passed"
  • <Relative url>
  • <launchParameters>
  • <entitlementKey>

AU2

  • launchMethod="OwnWindow"
  • moveOn=" Completed "
  • <Relative url>
  • <launchParameters>
  • <entitlementKey>

AU3

  • launchMethod="OwnWindow"
  • masteryScore="0.8501"
  • moveOn="CompletedAndPassed "
  • <Relative url>
  • <launchParameters>
  • <entitlementKey>

AU4

  • launchMethod="OwnWindow"
  • masteryScore="0.8501"
  • moveOn="CompletedOrPassed "
  • <Relative url>
  • <launchParameters>
  • <entitlementKey>

AU5

  • launchMethod="OwnWindow"
  • moveOn="NotApplicable"
  • <Relative url>
  • <launchParameters>
  • <entitlementKey>

Block1.1.1

Block1.1.1.1

AU6

  • launchMethod="OwnWindow"
  • moveOn=" Completed"
  • <Fully Qualified url>
  • <launchParameters>
  • <entitlementKey>

AUx

  • launchMethod="OwnWindow"
  • moveOn="NotApplicable"
  • <Relative url>
  • <launchParameters>
  • <entitlementKey>

Large Course Structure test (LMS) Test

  • Created by testsuite (Static or generated)
  • Has 1001 AU's
  • Fully Qualified URL
  • Zipped and XML only

Block1

AU1

  • launchMethod="OwnWindow"
  • moveOn=" Completed "
  • <url>
  • <launchParameters>
  • <entitlementKey>

…

AU1 1001

  • launchMethod="OwnWindow"
  • moveOn=" Completed "
  • <url>
  • <launchParameters>
  • <entitlementKey>

Testing Procedures

Small course testing

  •  TestSuite Generates a test course structure (XML)
    • oo (Initially) Static one to start
    • oo (Later) a parameter driven generation of course structure.
  •  (Human) Import that structure(s) into the LMS (admin user)
  •  (Human) Register multiple learners (admin User) – User 1, 2, 3….
  •  (Human) Access the course as a learner 1
    • oo Launch AU's in order
      •  Do Not Launch (AU5 – Not applicable)
    • oo Complete all AU's with minimum MoveOn Criterial
    • oo Single Entry
    • oo Final Check AU
      •  Run AU x after course complete (check for all satisfied)
      •  A wait period should be established
      •  AU x is Not Applicable
      •  Alternate Credentials to Access to LRS for LMS issued statements.
  •  (Human) Access the course as a learner 2
    • oo Launch AU's in order
    • oo Complete all AU's with MoveOn Criterial
    • oo Multiple Entry
    • oo Final Check AU
      •  Run AU x after course complete (check for all satisfied)
      •  A wait period should be established
      •  AU x is Not Applicable
      •  Alternate Credentials to Access to LRS for LMS issued statements.
  •  (Human) Access the course as a learner 3
    • oo Launch AU's in any order
    • oo Multiple Entry (Re-launching - Exiting and Re-entering AU)
    • oo Complete all AU's except AU6
      •  Partially Complete AU6 and re-enter
      •  Complete AU6
    • oo Final Check AU
      •  Run AU x after course complete (check for all satisfied)
      •  A wait period should be established
      •  AU x is Not Applicable
      •  Alternate Credentials to Access to LRS for LMS issued statements.

Large course testing

  •  TestSuite Generates a test course structure (XML)
    • oo (Initially) Static one to start
    • oo (Later) a parameter driven generation of course structure.
  •  (Machine) AU's (Testsuite) inspect LMS/LRS data
  •  (Human) Procedure for learner to follow thru the course(s).
    • oo Initial entry
    • oo Re-entry of AU's
  •  (Human) Modify course structure (admin user) and verify effect as a learner
    • oo Eg, MasteryScore, MoveOn, etc

Detail Procedure

Small course testing

  •  TestSuite Generates a test course structure (XML)
    • oo (Initially) Static one to start
    • oo (Later) a parameter driven generation of course structure.
  •  (Human) Import that structure(s) into the LMS (admin user)
  •  (Human) Register multiple learners (admin User) – User 1, 2, 3….
  •  (Human) Access the course as a learner 1
    • oo Launch AU0
      •  Inspect URL (for all N/V pairs)
      •  Get Auth Token (fetch)
      •  Inspect State API
      •  Issue Initialized Statement
      •  Send Complete Statement
      • 
      •  Do Not Launch (AU5 – Not applicable)
    • oo Complete all AU's with minimum MoveOn Criterial
    • oo Single Entry
    • oo Final Check AU
      •  Run AU x after course complete (check for all satisfied)
      •  A wait period should be established
      •  AU x is Not Applicable
      •  Alternate Credentials to Access to LRS for LMS issued statements.
  •  (Human) Access the course as a learner 2
    • oo Launch AU's in order
    • oo Complete all AU's with MoveOn Criterial
    • oo Multiple Entry
    • oo Final Check AU
      •  Run AU x after course complete (check for all satisfied)
      •  A wait period should be established
      •  AU x is Not Applicable
      •  Alternate Credentials to Access to LRS for LMS issued statements.
  •  (Human) Access the course as a learner 3
    • oo Launch AU's in any order
    • oo Multiple Entry (Re-launching - Exiting and Re-entering AU)
    • oo Complete all AU's except AU6
      •  Partially Complete AU6 and re-enter
      •  Complete AU6
    • oo Final Check AU
      •  Run AU x after course complete (check for all satisfied)
      •  A wait period should be established
      •  AU x is Not Applicable
      •  Alternate Credentials to Access to LRS for LMS issued statements.

Individual AU Test (Simple)

Scope:

  • Simple Launch (Initial and re-entrant)

Setup:

  • Course Structure
  • Fully qualified URL to content
  • LaunchData= AU specific parameters
  • Moveon=Completed
  1. 1. Add setup data – import course Structure
  2. 2. Register Learner in course, etc.
  3. 3.** Action: Launch AU**
  • oo Result: LaunchConsole (LMS)
    •  Launched Statement Issued
    •  Write State API Document LMS.launchdata
    •  Redirect Browser to AU URL +cmi5 Query string
  • oo Result: AU
    •  Parse Query String
    •  Call Fetch URL – Retrieve Authentication Token
      •  Call Fetch URL 2 nd Time to verify Error Issued
    •  Read State API Document LMS.launchdata (verify structure)
      •  Verify Launch Data Parameters from course structure are the same
    •  Write Initialized Statement (success)
    •  Present AU UI to learner (with results of checks)
      •  OK/Failed list
  1. 4.** Action: Exit AU (Button)**
    •  Write AU specific State API Document (state Id = AUtest, Registration, ActivityId, Actor)
    •  Write Terminated Statement (Success)
    •  Return To LMS (Use Return URL if provided or Close The AU window)
  1. 5.** Action: **** (re) Launch AU**
  • oo Result: LaunchConsole (LMS)
    •  Launched Statement Issued
    •  Write State API Document LMS.launchdata
    •  Redirect Browser to AU URL +cmi5 Query string
  • oo Result: AU
    •  Parse Query String
    •  Call Fetch URL – Retrieve Authentication Token
    •  Read State API Document LMS.launchdata (verify structure)
      •  Verify Launch Data Parameters from course structure are the same
    •  Write Initialized Statement (success)
    •  Read AU-Specific State API Document. (Verify contents same)
    •  Present AU UI to learner (with results of checks)
      •  OK/Failed list
      •  Previous Statements
  1. 6.** Action: Send "Completed" (Button)**
  • oo Result: AU
    •  Write Completed Statement (Success)
  1. 7.** Action: Exit AU (Button)**
  • oo Result: AU
    •  Write Terminated Statement (Success)
    •  Return To LMS (Use Return URL if provided or Close The AU window)
  • oo Result: LaunchConsole (LMS)
    •  Satisfied Statement Issued – At Course Level
    •  Verify Abandon Statement not issued for each session
      •  After a reasonable period of time

Individual AU Test (One Entrance - Passed)

Scope:

  • Simple Launch (ne launch pass)

Setup:

  • Course Structure
  • Fully qualified URL to content
  • LaunchData= AU specific parameters
  • Moveon=Passed
  1. 1. Add setup data – import course Structure
  2. 2. Register Learner in course, etc.
  3. 3.** Action: Launch AU**
  • oo Result: LaunchConsole (LMS)
    •  Launched Statement Issued
    •  Write State API Document LMS.launchdata
    •  Redirect Browser to AU URL +cmi5 Query string
  • oo Result: AU
    •  Parse Query String
    •  Call Fetch URL – Retrieve Authentication Token
      •  Call Fetch URL 2 nd Time to verify Error Issued
    •  Read State API Document LMS.launchdata (verify structure)
      •  Verify Launch Data Parameters from course structure are the same
    •  Write Initialized Statement (success)
    •  Present AU UI to learner (with results of checks)
      •  OK/Failed list
  1. 4.** Action: Send "Pass" (Button)**
  • oo Result: AU
    •  Write Passed Statement (Success)
  1. 5.** Action: Exit AU (Button)**
    •  Write AU specific State API Document (state Id = AUtest, Registration, ActivityId, Actor)
    •  Write Terminated Statement (Success)
    •  Return To LMS (Use Return URL if provided or Close The AU window)
  • oo Result: LaunchConsole (LMS)
    •  Satisfied Statement Issued – At Course Level
    •  Verify Abandon Statement not issued for each session
      •  After a reasonable period of time

Individual AU Test (Abandoned – AU crash After Launch)

Scope:

  • Simple Launch (Abandoned – AU crash After Launch)

Setup:

  • Course Structure
  • Fully qualified URL to content
  • LaunchData= AU specific parameters
  • Moveon=Completed
  1. 6. Add setup data – import course Structure
  2. 7. Register Learner in course, etc.
  3. 8.** Action: Launch AU**
  • oo Result: LaunchConsole (LMS)
    •  Launched Statement Issued
    •  Write State API Document LMS.launchdata
    •  Redirect Browser to AU URL +cmi5 Query string
  • oo Result: AU
    •  Parse Query String
    •  Call Fetch URL – Retrieve Authentication Token
      •  Call Fetch URL 2 nd Time to verify Error Issued
    •  Read State API Document LMS.launchdata (verify structure)
      •  Verify Launch Data Parameters from course structure are the same
    •  Write Initialized Statement (success)
    •  Present AU UI to learner (with results of checks)
      •  OK/Failed list
  1. 9.** Action: Close AU (Abnormally Exit)**
  2. 10.** Action: Re-Launch AU (new session)**
  • oo Result: LaunchConsole (LMS)
    •  Abandoned Statement Issued
    •  Launched Statement Issued
    •  Write State API Document LMS.launchdata
    •  Redirect Browser to AU URL +cmi5 Query string
  • oo Result: AU
    •  Parse Query String
    •  Call Fetch URL – Retrieve Authentication Token
      •  Call Fetch URL 2 nd Time to verify Error Issued
    •  Read State API Document LMS.launchdata (verify structure)
      •  Verify Launch Data Parameters from course structure are the same
    •  Write Initialized Statement (success)
    •  Present AU UI to learner (with results of checks)
      •  OK/Failed list
    • 
  1. 11.** Action: Exit AU (Button)**
    •  Write AU specific State API Document (state Id = AUtest, Registration, ActivityId, Actor)
    •  Write Terminated Statement (Success)
    •  Return To LMS (Use Return URL if provided or Close The AU window)
  • oo Result: LaunchConsole (LMS)
    •  Verify No Satisfied Statement
    •  Verify Abandon Statement not issued for second session
      •  After a reasonable period of time

Individual AU Test (2 AU Course - Abandoned)

Scope:

  • Simple Launch (abandoned)

Setup:

  • Course Structure – 2 AUs
  • Fully qualified URL to content
  • LaunchData= AU specific parameters
  • Moveon=Completed, Completed
  1. 1. Add setup data – import course Structure
  2. 2. Register Learner in course, etc.
  3. 3.** Action: Launch AU 1**
  • oo Result: LaunchConsole (LMS)
    •  Launched Statement Issued
    •  Write State API Document LMS.launchdata
    •  Redirect Browser to AU URL +cmi5 Query string
  • oo Result: AU
    •  Parse Query String
    •  Call Fetch URL – Retrieve Authentication Token
      •  Call Fetch URL 2 nd Time to verify Error Issued
    •  Read State API Document LMS.launchdata (verify structure)
      •  Verify Launch Data Parameters from course structure are the same
    •  Write Initialized Statement (success)
    •  Present AU UI to learner (with results of checks)
      •  OK/Failed list
  1. 4.** Action: Close AU 1 (Abnormally Exit)**
  2. 5.** Action: Launch AU2 (new session)**
  • oo Result: LaunchConsole (LMS)
    •  Abandoned Statement Issued (for AU1 session)
    •  Launched Statement Issued (AU2 Session)
    •  Write State API Document LMS.launchdata
    •  Redirect Browser to AU URL +cmi5 Query string
  • oo Result: AU
    •  Parse Query String
    •  Call Fetch URL – Retrieve Authentication Token
      •  Call Fetch URL 2 nd Time to verify Error Issued
    •  Read State API Document LMS.launchdata (verify structure)
      •  Verify Launch Data Parameters from course structure are the same
    •  Write Initialized Statement (success)
    •  Present AU UI to learner (with results of checks)
      •  OK/Failed list
    • 
  1. 6.** Action: Exit AU (Button)**
    •  Write AU specific State API Document (state Id = AUtest, Registration, ActivityId, Actor)
    •  Write Terminated Statement (Success)
    •  Return To LMS (Use Return URL if provided or Close The AU window)
  • oo Result: LaunchConsole (LMS)
    •  Verify No Satisfied Statement
    •  Verify Abandon Statement not issued for second session
      •  After a reasonable period of time

Individual AU Test (Terminate – Then write more)

Scope:

  • )

Setup:

  • Course Structure (1 AU)
  • Fully qualified URL to content
  • LaunchData= AU specific parameters
  • Moveon=Completed
  1. 1. Add setup data – import course Structure
  2. 2. Register Learner in course, etc.
  3. 3.** Action: Launch AU**
  • oo Result: LaunchConsole (LMS)
    •  Launched Statement Issued
    •  Write State API Document LMS.launchdata
    •  Redirect Browser to AU URL +cmi5 Query string
  • oo Result: AU
    •  Parse Query String
    •  Call Fetch URL – Retrieve Authentication Token
      •  Call Fetch URL 2 nd Time to verify Error Issued
    •  Read State API Document LMS.launchdata (verify structure)
      •  Verify Launch Data Parameters from course structure are the same
    •  Write Initialized Statement (success)
    •  Present AU UI to learner (with results of checks)
      •  OK/Failed list
  1. 4.** Action: AU Terminate Button (Issue Terminate)**
  • oo Result: AU
    •  Issue Terminate Statement
    •  Then – Issue Completed Statement (after reasonable time –as defined in se)
  1. 5.** Action: Re-Launch AU (new session)**
  • oo Result: LaunchConsole (LMS)
    •  Abandoned Statement Issued
    •  Launched Statement Issued
    •  Write State API Document LMS.launchdata
    •  Redirect Browser to AU URL +cmi5 Query string
  • oo Result: AU
    •  Parse Query String
    •  Call Fetch URL – Retrieve Authentication Token
      •  Call Fetch URL 2 nd Time to verify Error Issued
    •  Read State API Document LMS.launchdata (verify structure)
      •  Verify Launch Data Parameters from course structure are the same
    •  Write Initialized Statement (success)
    •  Present AU UI to learner (with results of checks)
      •  OK/Failed list
    • 
  1. 6.** Action: Exit AU (Button)**
    •  Write AU specific State API Document (state Id = AUtest, Registration, ActivityId, Actor)
    •  Write Terminated Statement (Success)
    •  Return To LMS (Use Return URL if provided or Close The AU window)
  • oo Result: LaunchConsole (LMS)
    •  Verify No Satisfied Statement
    •  Verify Abandon Statement not issued for second session
      •  After a reasonable period of time

_Disclaimers/Limits_

Because compliance cannot be verified – it does not mean that an LMS (or AU) is not cmi5 conformant.

Test suite AU failure to execute due to software/environment conflicts…

"** Reasonable Time" set by TestSuite parameters****. (Procedure will specify)**

Rationale: How long a learner user would expect to wait for LMS corrective action due to AU failure - up to 3 seconds