Tests - campsych/concerto-platform GitHub Wiki

A Test is an object that contains execution logic. This is the brain of your web app. Any given test can take a set of input parameters, perform a set of tasks and then output a set of return variables. These three components are described in detail below as Test Inputs, Test Logic and Test Outputs respectively.

• Accessibility – Describes who can view and edit an object in the administration panel. This can be one of the following:
  • private – Can only be accessed by owner of the object, or super admins.
  • group – Can be accessed by owner, users belonging to the same group as the object, or super admins.
  • public – Can be accessed by anyone logged in to the administration panel.
• Archived – Archived objects will not be selectable from test wizard parameters.
• Description – Text describing the purpose of the object. This is visible as contextual information when using this object.
• Groups – Groups that this object belongs to. Used in conjunction with Accessibility.
• Name – Unique, human readable identifier, that will be used to reference the test in your test logic.
• Owner – Object owner. Used in conjunction with Accessibility.
• Type – Describes the type of logic used. This can be one of the following:
  • code – the test logic is comprised of plain R code.
  • wizard – the test logic is determined by a test wizard. Wizards can be used to override the default values of test parameters.
  • flow – Test logic is scripted visually using the flow chart.
• URL name – Unique string that will be used in URL to launch this test.
• Visibility – Defines if and from where this test can be run. This can be one of the following:
  • regular – Can be launched directly from URL.
  • subtest – Can only be run from within another test, and cannot be launched directly from URL
  • featured – Can be launched directly from URL and will also be listed in ‘available tests’ dropdown on the main domain where you have Concerto platform installed.

Input parameters can be passed to a Concerto test and used in test logic. The test input section is not available for wizard-based tests. Each input parameter consists of the following properties:

• Name – Identifier that will be used to reference the input parameter.
• Description – Text describing the purpose of the input parameter.
• URL – If checked, it means that this input parameter value can be set through the URL (using a URL parameter with same name). If unchecked, it will not allow this parameter to be set through the URL. This only applies for for tests that can be launched directly through the URL and not to sub-tests.
• Default value – Default string value to use for this parameter when no value has been specified. Defaults to NA.

Test logic section will be different depending on the test Type selected.

For code-based tests, the test logic will be plain R code that should be executed when the test is run. Input parameters can be accessed directly in test logic code. You can set values for your return variables by just declaring a variable with the same name in your code.

Typical use case:

• reusable functions/subroutines

In wizard-based tests you are not able to change the logic directly. What you can do is, set the value of exposed test wizard parameters, that are test input parameters of test wizard source test, but with extra graphic user interface to simplify setting their value (and make it less error prone).
Every test wizard has a source test that it is built on top of. Wizard-based tests will use their source test’s input parameters, logic and return variables at execution.
Each test wizard parameter can belong to a step in the test wizard and each wizard can be split into multiple steps.

Typical use case:

• parametrized tests that share the same test logic, but implementations differs in parameter values (for example multi-language versions of the same test but with different item banks or norms)
• some starter content nodes are effectively wizards whose source test is in R code

In flow-based tests you can script your logic visually by connecting elements in the flowchart. You can do this using following nodes, which are the building blocks of flow-based tests:

• Nodes
  Each node is a test that can have its input parameters, return variables and branch variables exposed through test node ports. When flow execution reaches a node, it passes its input parameter to the underlying test logic, executes it, sets return variables ports values and then selects a branch port to use for further flow execution. This means you can string nodes together in the flowchart to accomplish any tasks that you want your test to perform.

Special types of nodes:

• test start – Always present, this is the entry node for execution and always come first in flow. When a flow-based test has input parameters defined, these will be available through the return variable ports on the test start node.
• test end – Always present, this is the end node for execution and always come last in flow. When a flow-based test has return variables defined, these will be available through the input parameters ports on the test end node. If you want to use the return variables of a given flow-based test elsewhere, then you need to ensure that this test end node is executed i.e. that whatever comes last in your test is connected to this node.

Ports
Ports are the endpoints in data and execution flow between nodes. All nodes have ports and they come in the following types:

• execution ‘in’ port
• execution ‘out’ port (branch)
• input parameter port
• return variable port

Connections
Each connection is made from one port to another. There are two possible connection types:

• execution – Made from execution ‘out’ port to execution ‘in’ port. This connection tells the flow what to do after a given node has finished executing its code. This is done by connecting the execution ‘out’ port of the first node to the execution ‘in’ port of the second node.
• data – Made from a return variable port to an input parameter port. This can be used to pass variables between nodes i.e. from the return variable port of node A to the input parameter port of node B. The value of the input parameter port will be set to the return variable when node B is executed. Data connections can also have a return function defined, which is an R code function that performs operations on the return variable from node A before setting it as the input parameter of node B.

Typical use case:

• top-most test execution logic

Test Outputs

This section does not apply to wizard-based tests. The outputs from other tests are split into two categories:

• Return variables – Variables containing test results
• Branches – Execution path that describes what should happen next

Similarly to Input parameters, Return variables and Branches are defined by the following properties:

• Name – Identifier that will be used to reference the output.
• Description – Text describing the role of the output.
• Default value – Default string value to use when no value has been specified for the output. Defaults to NA. The default value property does not do anything for Branches.

Error logs

Concerto has an in-built error log that will list both test logic errors (R exceptions) and template errors (javascript exceptions). Tests need to be run directly (and not as a subtest) in order to log errors.