How to Run the Test Suite Locally - xspec/xspec GitHub Wiki

XSpec is automatically tested via GitHub Actions, Azure Pipelines and AppVeyor every time a pull request is submitted or merged into the master branch. Contributors can also run the test suite locally before submitting a pull request; doing so usually avoids breaking tests and speeds up the review process.

• Installing Prerequisites
• (Optional) Setting XSpec Test Environment
• Installing Dependencies
• Running Test Suites
• Tips

As always, Java Runtime Environment must be installed.

sudo apt install default-jre

For Linux/macOS, you need Bats v1.8.0+:

sudo npm install -g bats

brew install bats-core

curl command line tool is used for downloading the test dependencies.

sudo apt install curl

tar command line tool is used for extracting the test dependencies.

XSpec is tested against multiple sets of dependencies. If the default set is sufficient for your local testing, you can skip this step.

If you need to test locally against a particular set of dependencies (e.g., you are upgrading the Saxon version used in testing and, therefore, need to run that particular version), this step configures that set. Do this step whenever you open a new terminal (or command prompt).

1. Look in the test/ci/env/ directory for the .env files other than global.env. These text files define the different test environments. In that directory, the README.md file has some notes about the syntax the .env files use.
2. Find the .env file that corresponds to the set of dependencies you want to use for your local testing.
3. Set the XSPEC_TEST_ENV environment variable to the filename stem of the .env file you want to use. For instance, enter export XSPEC_TEST_ENV=saxon-10 on Linux/macOS or set XSPEC_TEST_ENV=saxon-10 on Windows.

After installing the prerequisites, you need to install the dependencies.

• Apache Ant
• Saxon-HE (Java)
• Apache XML Resolver (part of Apache XML Commons)
• XML Calabash
• BaseX

Installing the dependencies is automated. They are installed in a temporary directory and some volatile environment variables are set for the current terminal (or command prompt). Since they are volatile, you need to do this installation whenever you open a new terminal (or command prompt).

Run test/ci/install-deps.sh with source:

$ source test/ci/install-deps.sh
Setting up saxon-9-9
...

Check the installation state with test/ci/print-env.sh:

$ test/ci/print-env.sh

=== Print Java version
java version "1.8.0_151"

=== Print Ant version
Apache Ant(TM) version 1.10.7 compiled on September 1 2019

=== Print Saxon version
SAXON-HE 9.9.1.7J from Saxonica (build 030415)

=== Check XML Calabash
XML Calabash version 1.1.30, an XProc processor.

=== Print XML Resolver version
XmlResolver 1.2

=== Check BaseX
BaseX 9.3.1 [Standalone]

=== Check BaseX server start and stop
HTTP Server was started (port: 8984).
HTTP Server was stopped (port: 8984).

=== Print Bats version
Bats 0.4.0

=== Print environment variables
XMLCALABASH_JAR=/tmp/xspec/xmlcalabash/xmlcalabash-1.2.5-99/xmlcalabash-1.2.5-99.jar
XML_RESOLVER_JAR=/tmp/xspec/xml-resolver/resolver.jar
BASEX_JAR=/tmp/xspec/basex/basex/BaseX.jar
SAXON_JAR=/tmp/xspec/saxon/saxon9he.jar

Run test\ci\install-deps.cmd:

C:\xspec>test\ci\install-deps.cmd
Setting up saxon-9-9
...

Check the installation state with test\ci\print-env.cmd:

C:\xspec>test\ci\print-env.cmd

=== Print Java version
openjdk version "1.8.0_222"

=== Print Ant version
Apache Ant(TM) version 1.10.7 compiled on September 1 2019

=== Print Saxon version
SAXON-HE 9.9.1.7J from Saxonica (build 030415)

=== Check XML Calabash
XML Calabash version 1.1.30, an XProc processor.

=== Print XML Resolver version
XmlResolver 1.2

=== Check BaseX
BaseX 9.3.1 [Standalone]

=== Check BaseX server start and stop
HTTP Server was started (port: 8984).
HTTP Server was stopped (port: 8984).

=== Print environment variables
BASEX_JAR=C:\Users\VSSADM~1\AppData\Local\Temp\xspec\basex\basex\BaseX.jar
SAXON_JAR=C:\Users\VSSADM~1\AppData\Local\Temp\xspec\saxon\saxon9he.jar
XMLCALABASH_JAR=C:\Users\VSSADM~1\AppData\Local\Temp\xspec\xmlcalabash\xmlcalabash-1.2.5-99\xmlcalabash-1.2.5-99.jar
XML_RESOLVER_JAR=C:\Users\VSSADM~1\AppData\Local\Temp\xspec\xml-resolver\resolver.jar

After installing the prerequisites and the dependencies, you should now be able to run the XSpec test suites on your machine. Go to the xspec directory on the branch you want to test and run the following scripts for executing the full test suite:

• Bats tests
• XSpec tests
• End-to-end tests

This test suite ensures that XSpec behaves as expected under specific conditions. If you are missing a dependency, the test suite should be able to tell you what is missing or skip some tests.

$ test/run-bats.sh
...
75 tests, 0 failures

C:\xspec>test\run-bats.cmd
=== START TEST CASES ================================================
...
=== END TEST CASES ==================================================
EXIT_CODE=0

This test suite ensures that test/*.xspec files produce no Failure result on Schematron/XQuery/XSLT, which ensures there is no regression in the core logic of XSpec (src/compiler in particular). See README for details.

It should end with BUILD SUCCESSFUL.

$ test/run-xspec-tests-ant.sh
Buildfile: .../xspec/test/ant/build.xml
...
BUILD SUCCESSFUL
Total time: 3 minutes 55 seconds

C:\xspec>test\run-xspec-tests-ant.cmd
...
Buildfile: C:\xspec\test\ant\build.xml
...
BUILD SUCCESSFUL
Total time: 1 minute 3 seconds

These test suites ensure that test\end-to-end\cases\*.xspec and test\end-to-end\cases-coverage\*.xspec files produce the report files as expected on Schematron/XQuery/XSLT. See README for details.

Each one should end with BUILD SUCCESSFUL.

$ test/end-to-end/run-e2e-tests.sh
Buildfile: .../xspec/test/end-to-end/ant/run-e2e-tests/build.xml
...
BUILD SUCCESSFUL
Total time: 2 minutes 31 seconds

$ test/end-to-end/run-e2e-coverage-tests.sh
Buildfile: .../xspec/test/end-to-end/ant/run-e2e-tests/build.xml
...
BUILD SUCCESSFUL
Total time: 23 seconds

C:\xspec>test\end-to-end\run-e2e-tests.cmd
...
Buildfile: C:\xspec\test\end-to-end\ant\run-e2e-tests\build.xml
...
BUILD SUCCESSFUL
Total time: 52 seconds

C:\xspec>test\end-to-end\run-e2e-coverage-tests.cmd
...
Buildfile: C:\xspec\test\end-to-end\ant\run-e2e-tests\build.xml
...
BUILD SUCCESSFUL
Total time: 23 seconds

• Windows virtual machines for various hypervisers including VirtualBox are available for testing. They are valid for 90 days and can be downloaded at the Microsoft website.

• Do not try to run all the test suite commands in parallel in multiple shell windows, as the commands can interfere with each other.

• Do not press Ctrl+c before the tests finish running. That could cause intermediate files to remain in your workspace, which could cause your next test run to fail.

• If you have in-progress test files with file names ending with .xspec, they might get run as part of the test suite. If you expect them to fail and don't want those failures to interfere with the test suite results, you can mark failing scenarios as pending or rename the files to avoid ending with .xspec.

• If you want to run some of the Bats test cases and skip all the other cases, you can set --filter parameter of run-bats.sh or run-bats.cmd:

  Linux/macOS

  $ test/run-bats.sh --filter "CUSTOM|HOME"
   ✓ XSPEC_HOME
   ✓ XSPEC_HOME is not a directory
   ✓ XSPEC_HOME seems to be corrupted
   ✓ SAXON_CP has precedence over SAXON_HOME
   ✓ invoking xspec using SAXON_HOME finds Saxon jar and XML Catalog Resolver jar
   ✓ invoking xspec for XSLT with SAXON_CUSTOM_OPTIONS
   ✓ invoking xspec for XQuery with SAXON_CUSTOM_OPTIONS
  
  7 tests, 0 failures

  Windows

  C:\xspec>test\run-bats.cmd --filter "CUSTOM|HOME"
  7 test case(s) (Filter: "CUSTOM|HOME")
  === START TEST CASES ================================================
  CASE #1: XSPEC_HOME
  ...
  ...PASS
  CASE #2: XSPEC_HOME is not a directory
  ...
  ...PASS
  CASE #3: XSPEC_HOME seems to be corrupted
  ...
  ...PASS
  CASE #4: SAXON_CP has precedence over SAXON_HOME
  ...
  ...PASS
  CASE #5: invoking xspec using SAXON_HOME finds Saxon jar and XML Catalog Resolver jar
  ...
  ...PASS
  CASE #6: invoking xspec for XSLT with SAXON_CUSTOM_OPTIONS
  ...
  ...PASS
  CASE #7: invoking xspec for XQuery with SAXON_CUSTOM_OPTIONS
  ...
  ...PASS
  === END TEST CASES ==================================================
  EXIT_CODE=0

• If you want to inspect temporary files after running Bats, you can set --no-tempdir-cleanup command line option and check the temporary directory:

  $ test/run-bats.sh --no-tempdir-cleanup
  ...
  BATS_RUN_TMPDIR: /tmp/bats-run-qqggFd

  This option is not available on Windows.

• XSpec tests and End-to-end tests run in parallel, and interspersed log messages can make it difficult to tell which test caused an Ant failure. When investigating a failure in your local workspace, consider temporarily specifying thread.count=1 Ant property:

  Linux/macOS

  test/run-xspec-tests-ant.sh -Dthread.count=1

  test/end-to-end/run-e2e-tests.sh -Dthread.count=1

  test/end-to-end/run-e2e-coverage-tests.sh -Dthread.count=1

  Windows

  test\run-xspec-tests-ant.cmd -Dthread.count=1

  test\end-to-end\run-e2e-tests.cmd -Dthread.count=1

  test\end-to-end\run-e2e-coverage-tests.cmd -Dthread.count=1

  Oxygen

  In the Transformation Scenarios pane, duplicate ANT (with Saxon 9 EE XSLT support) and set thread.count to 1 in the Parameters tab.

  That way, XSpec tests will run sequentially.

• If you have a log from XSpec tests with interspersed messages and have trouble finding out which specific XSpec test failed, look for the message about test/ant/worker/build-worker.xml in the stack of error messages. If it references a line number in this generated file, that line should provide the path to the failing test.

• If you want to run only some of the test cases of XSpec tests or End-to-end tests, you can set xspecfiles.dir.url.query Ant property:

  Linux/macOS

  test/run-xspec-tests-ant.sh -Dxspecfiles.dir.url.query="select=issue-*.xspec"

  test/end-to-end/run-e2e-tests.sh -Dxspecfiles.dir.url.query="select=issue-*.xspec"

  test/end-to-end/run-e2e-coverage-tests.sh -Dxspecfiles.dir.url.query="select=issue-*.xspec"

  Windows

  test\run-xspec-tests-ant.cmd -Dxspecfiles.dir.url.query="select=issue-*.xspec"

  test\end-to-end\run-e2e-tests.cmd -Dxspecfiles.dir.url.query="select=issue-*.xspec"

  test\end-to-end\run-e2e-coverage-tests.cmd -Dxspecfiles.dir.url.query="select=issue-*.xspec"

  xspecfiles.dir.url.query Ant property is used when scanning the test case directory. It works as a query parameter of URI passed to Saxon's fn:collection() implementation.

• Debugging XProc is not an easy task, as it lacks a live debugger. To inspect intermediate artifacts going through the pipeline, you can use x:log step implemented in src/harnesses/harness-lib.xpl:

  Linux/macOS

  java -cp xmlcalabash-1.5.4-100.jar:slf4j-simple-1.7.36.jar \
       com.xmlcalabash.drivers.Main \
       -i source=tutorial/escape-for-regex.xspec \
       -p xspec-home=file:/tmp/xspec/ \
       -p log-indent=/tmp/log.txt \
       src/harnesses/saxon/saxon-xslt-harness.xproc

  Windows

  java -cp "xmlcalabash-1.5.4-100.jar;slf4j-simple-1.7.36.jar" ^
       com.xmlcalabash.drivers.Main ^
       -i source=tutorial/escape-for-regex.xspec ^
       -p xspec-home="file:///C:/xspec/" ^
       -p log-indent=file:///C:/my/log.txt ^
       src\harnesses\saxon\saxon-xslt-harness.xproc

  In this example, you'll get /tmp/log.txt (Linux/macOS) or C:\my\log.txt (Windows) file containing the artifact at <x:log if-set="log-indent">.