Understanding the Testing Structure

The total collection, order, and grouping of tests in an Inferno instance is called a module. A module is comprised of "test sets", each of which contains a view and a collection of test groups. The view determines how the tests are presented to the user and the test groups are ordered collections of sequences that share a common theme.

A sequence represents some aspect of the API you want to test and is comprised of smaller "tests". For example, the Conformance Statement sequence is meant to test if the specified server has a properly formatted conformance statement. It does this by testing if the supplied conformance statement describes SMART on FHIR core capabilities, states JSON support, and lists supported Argonaut profiles. Each of these tests is comprised of several "assertions", which are just checks to see if something is true. The tests for this example can be seen in argonaut_conformance_sequence.rb.

There are currently only two view built into Inferno, "default" and "guided". The default view is meant to be a more free-form way of performing tests where you are able to run sequences in any order. The guided view is meant to be used in a certification context where there's a defined order to which you run the sequences.

The following represents the hierarchy of testing entities within Inferno:

Module
  Test Sets
    Test Groups
      Sequences
        Tests
          Assertions

Creating a Sequence File

A sequence is defined by a ruby class that extends the SequenceBase class. Each sequence contains properties that describe what the test is for and what is required for it to run.

The following is a list of sequence properties and what they are for:

Property	Description
title	What appears at the top of each sequence.
description	A short description of the sequence. Appears directly underneath the title.
details	An optional detailed description of a sequence. This will appear under the "Overview" tab in the sequence if provided.
test_id_prefix	A unique identifier for the sequence.
requires	List of variables from the testing instance that are needed to run the sequence. They will appear as inputs in a modal popup after trying to run the sequence.
defines	List of variables that this sequence provides values for. These are effectively outputs of the sequence that gets saved to the testing instance.
optional	If specified, makes the sequence not count against the overall result of the test group.
show_uris	If specified, shows the launch and redirect URI's in the modal popup when run.
conformance_supports	Optional - Resource that the sequence is testing. If the conformance statement does not indicate support for this resource, a warning message is displayed in the sequence.

Example of a custom sequence:

class CustomSequence < SequenceBase
  title 'Your Custom Title'
  description 'Describe your sequence here'
  test_id_prefix 'CS'
  required :id_token, :client_id
  defines :oauth_introspection_endpoint
  details 'Add details to be displayed in the overview tab. This property is optional'
  
  # tests go here
end

You can also extend sequences to create child sequences. Child sequences will contain all the tests of the parent, but will also have its own properties and tests. You must include extends_sequence ParentSequence in the child for this to work. See argonaut_conformance_sequence.rb as an example.

Adding Tests to a Sequence

Tests can be added to a sequence through the function "test". Each test contains some metadata about itself followed by a series of "assertions". An assertion simply checks to see if some value evaluates to true. If one assertion evaluates to false, then the test ends and the rest of the assertions are not run. Between the metadata and the assertions can be any valid Ruby code. A common example is the use of the @client variable, which is a which is a fhir_client initialized with an expected FHIR version and endpoint. See argonaut_patient_read_only_sequence.rb for an example use of @client along with assertions.

Example of a custom test:

test 'Short description of what this assertion tests for' do 
   metadata {
      id '01'
      link 'You can specify a link here display when the user opens the test result. This is optional'
      desc 'A longer description of what this assertion tests for'
   }
   # your code here
   assert something_that_should_be_true, 'This message displays on failure'
end

Creating a Module File

Once you have the sequences you want to add, you can add them to an existing module file or create a new one. Modules are defined by configuration files in the lib/app/modules directory. Create a new file (make sure it ends in "_module.yml") here to add your own module.

The module file structures the sequences into groups and describes some ways they are presented to the user, but all the actual testing is done in the sequence files you created above. Each module will have a title, description, FHIR version, default test set, and a group of test sets.

You are also able to override a sequence's title and description or add default values for its inputs in the module file.

Here is an example module file:

name: example_module
title: Title you want displayed on the landing page
description: This is a long description of the tests to appear under the title. 
fhir_version: dstu2
hide_optional: true
default_test_set: test_set_1
test_sets:
  test_set_1:
    view: guided
    tests:
      -name: Test Group 1
       overview: Overview for test group 1.
       input_instructions: instructions for inputs needed for test group 1
       sequences:
         -sequence: sequence1
          title: Override sequence title
         -sequence2
      -name: Test Group 2
       overview: Overview for test group 2
       input_instructions: instructions for inputs needed for test group 2
       sequences:
         -sequence: sequence3
          description: Override the sequence description
          variable_defaults:
            variable: add a default value for the variable specified when running this sequence
       lock_variables:
         - locked_variable1
         - locked_variable2
       run_all: true

Adding a New Module to Inferno

In the config.yml file, there is a section in the end for modules. Simply add your module to the list to be able to select it when launching Inferno.