sample.json CI Test Object

Within each sample.json file there is a "CI test object"(required) that contains tests to be performed by the sample CI (continuous integration) testing system. This object is a place to store a sequence of commands that, in the simplest case, can be used to validate that a sample builds and runs. This document describes how to add this "CI test object" to your sample.json file so it can be automatically tested by the sample CI testing system.

NOTE: All samples are tested by the CI testing system using the command-line on a Linux, Windows or Mac (aka Darwin) machine, as specified by the CI test object. Your sample is not automatically tested within the context of an IDE or editor, such as Visual Studio, Eclipse or VSCode. Therefore, it is assumed that command-line test results will be consistent with and reflect IDE results; if that is not the case, the samples CI test system may be inadequate for confirming your sample is "good."

A description of the sample.json file and its attributes can be found in the sample.json Specification wiki page.

TODO: Describe the underlying test machines (which OS, which OS version, which shell, which installed core and dev tools, etc.).

"ciTests" Object

Each sample.json file is expected to contain a ciTests object that contains instructions for running that sample's CI automated tests. The CI test system operates under the assumption that a failure will be signaled via a non-zero return code from a sequence of command-line steps. These command-line test steps are expected to be provided by the sample author.

The basic layout of the ciTests object is:

"ciTests": {
  "linux": [
    { "id": "test_id_1", "env": [ ], "steps": [ ] },
    { "id": "test_id_2", "env": [ ], "steps": [ ] },
    { "..." }, "..." 
  ],
  "windows": [ 
    { "id": "test_id_1", "env": [ ], "steps": [ ] },
    { "id": "test_id_2", "env": [ ], "steps": [ ] },
    { "..." }, "..." 
  ],
  "darwin": [ 
    { "id": "test_id_1", "env": [ ], "steps": [ ] },
    { "id": "test_id_2", "env": [ ], "steps": [ ] },
    { "..." }, "..." 
  ]
}

The "ciTests" object contains three OS elements named:

• "linux"
• "windows"
• "darwin"

Each OS element is an array of objects. Each object within those OS arrays contains:

• optional "id" string
• optional "env" array of strings
• required "steps" array of strings

Only those operating systems that are supported by the sample need to be present in the ciTests object. For example, a sample that does not support building and running on macOS does not need to provide a "darwin" array.

The "id" String

The optional "id" string is an arbitrary string that serves to mark the output of the CI tests performed by that object of "env" and "steps" arrays. Choose a meaningful and unique name so that you can more easily review the CI output for your sample, in the event one of the CI tests fails.

The "env" Array of Strings

The optional array of environment strings may contain a simple environment variable assignment or a script that will be run before the "steps" tests are executed. The "env" commands are run before the associated CI tests. The "env" commands are guaranteed to be executed in the order they are presented in the "env" array.

For example, on Linux and macOS:

  "env": [
    "export TEST=test",
    "export A=1",
    "source my-env-script.sh"
  ]

For example, on Windows:

  "env": [
    "set TEST=test",
    "set A=1",
    "call my-env-script.bat"
  ]

The "steps" Array of Strings

The required array of tests may be any valid command available on the OS being tested. The "steps" commands are run after the optional "env" commands have been executed. The "steps" commands are guaranteed to be executed in the order they are presented in the "steps" array.

For example, on Linux and macOS:

  "steps": [
    "rm -rf temp_dir || true"
    "mkdir -p build ; cd build ; cmake ..",
    "make all",
    "make run",
    "make clean"
  ]

Notice the use of || true in the the rm step in the snippet above. This is being done in order to force a "pass" result from that command. This would be done if you have a step in your tests for which the outcome (pass or fail) is not relevant and you do not want to have the test fail due to a fail at this step.

For example, on Windows:

  "steps": [
    "rmdir /s/q temp_dir"
    "mkdir build ; cd build ; cmake ..",
    "nmake all",
    "nmake run",
    "nmake clean"
  ]

How Sample CI Tests are Run

The "env" and "steps" commands are collected into a single script/batch file, which is then run on the CI test machine. The provided "env" commands are always placed before the "steps" commands in that script/batch file. The order of the "env" and "steps" arrays are guaranteed to be executed in the order they are presented in each array, where all "env" commands always execute before the "steps" commands.

A CI test stops when a "steps" command returns a non-zero error code. If an "env" command returns a non-zero error code, it will be ignored. Thus, a test passes only if all commands defined by the "steps" array return a zero error code, the "env" return codes do not affect the outcome of the CI test.

NOTE: The oneAPI top-level setvars.[sh|bat] script will be automatically sourced prior to running the combined "env" and "steps" script. Thus, there is no reason to include a reference to the setvars.[sh|bat] environment setup script in either your "env" or "tests" arrays.

Each test object ({ "id": "", "env": [ ], "steps": [ ] },) is is provided a fresh sample project and in all test commands are executed in a fresh shell. The current working directory (CWD) for your "env" and "steps" commands is the "root" of the directory of that fresh sample project. All files and folders that may be created by your test commands should be limited to the folders within the sample project's folders, in order to guarantee there are not after effects for subsequent tests.

Makefile Example on Linux or Mac

Makefile Example on Windows

CMake Example on Linux or Mac

CMake Example on Windows

Visual Studio Project File Example on Windows

Example Combined "ciTest" Object

"ciTests": {
  "linux": [
    {
      "id": "cpu_usm",
      "env": [
        "export TEST=test",
        "export A=1",
        "source my-env-script"
      ],
      "steps": [
        "mkdir -p build ; cd build ; cmake ..",
        "make all",
        "make run",
        "make clean"
      ]
    },
    {
      "id": "cpu_buffers",
      "steps": [
        "rm -rf temp_dir || true",
        "make build_buffers",
        "make run_buffers",
        "make clean"
      ]
    },
    {
      "id": "fpga_emu_buffers",
      "steps": [
        "make clean -f Makefile.fpga", "make fpga_emu -f Makefile.fpga",
        "make run_emu -f Makefile.fpga"
      ]
    },
    {
      "id": "fpga_reports_buffers",
      "steps": [
        "make clean -f Makefile.fpga",
        "make report -f Makefile.fpga"
      ]
    }
  ],
  "windows": [
    {
      "id": "cpu_usm",
      "env": [
        "set TEST=test",
        "set A=1"
      ],
      "steps": [
        "nmake -f Makefile.win",
        "nmake -f Makefile.win run",
        "nmake -f Makefile.win clean"
      ]
    },
    {
      "id": "fpga_emu_buffers",
      "steps": [
        "nmake -f Makefile.win.fpga",
        "nmake -f Makefile.win.fpga run",
        "nmake -f Makefile.win.fpga clean"
      ]
    }
  ]
}