Language Tests - NetLogo/NetLogo GitHub Wiki

We have a modest external DSL that makes it easy to write little self-contained tests that test the behavior of individual NetLogo language constructs, as well as whole models and extensions.

in these directories:

• test/commands/*.txt
• test/reporters/*.txt
• models/test/*.txt
• extensions/*/tests.txt

On the headless branch, the first two are inside a resources directory.

To run all command or reporter or model or extension tests from sbt:

tc
tr
tm
te

You can run a single group of tests with, for example:

tc InCone
tr Numbers
tm Fire
te array

These commands also support running a single test, for example:

tr Numbers::Sqrt3

All of the language tests are run as part of the sbt commands test and slow:test. tr and tc are part of medium:test. All are run by the main.yml ile which runs by GitHub Actions (see Continuous integration).

The tests for any extension can be run within the main NetLogo build just by putting the extension directory under extensions and using te from the NetLogo root directory.

Some extensions have been updated so that the tests can be run within the extension's own build, without having to build NetLogo. At least two extensions have this: matrix and nw. It would be nice if all the extension builds were updated in this way, probably by updating the NetLogo extension plugin to make it automatic.

The shell command to run the tests from within the extension repository is sbt test.

When you use =>, the value on the right (the expected value of the reporter on the left), must be a constant: a number, string, list, true/false, or nobody.

Or, if you expect a runtime error to occur, use ERROR, e.g.:

O> set glob1 foo => ERROR REPORT must be immediately inside a TO-REPORT.
[color] of glob1 => ERROR That mouse is dead.

For code that shouldn't even compile, say COMPILER ERROR instead, e.g.:

 O> __ignore value-from turtle 0 [color] => COMPILER ERROR Nothing named VALUE-FROM has been defined

If you want to break a statement up over multiple lines, add a forward-slash (/) at the end of each line that you want to continue onto the next. Do not put a slash on the last line of your statement, and do not put comments (#) inside your multi-line statement.

MultiLineTest
  to-report setup-code /
    create-turtles 100 [ fd 100 ] /
    ask turtles [ set label "hello" ] /
    report mean [pxcor] of turtles /
  end
  O> precision setup-code 0 => 0

Tests that end with _2D will be run in a 2D environment only, tests that end with _3D will be run in a 3D environment only, all other tests will be run in both environments.

Tests that start with Generator will only be run when the code generator is enabled. Tests that start with NoGenerator will only be run when the code generator is disabled. Other tests run regardless.

When you write tests that write files you should be careful to use a unique file name so that tests may run in parallel without interfering with each other. See test/commands/AgentsetImport or test/commands/File for examples.

Keep these simple. Don't use any of the special command-test features described below.

Nothing in reporter tests does anything with agents or agentsets. If you want to test agent stuff, use command tests, not reporter tests.

Nothing in reporter tests does anything involving randomness, since there's no way to set the random seed. Use command-tests for that.

You can define a procedure as part of a test.

Global variables glob1, glob2, and glob3 are available for use, plus a turtle variable tvar and a patch variable pvar.

Breeds mice and frogs are available. Both breeds have an age variable. Mice also have fur. Frogs also have spots. Also available are two link breeds, directed-links (with variable lvar) and undirected-links (with variable weight).

Two plots are available, plot1 and plot2. Both have two permanent pens named pen1 and pen2.

Command tests are run two times:

• once normally
• once with each command wrapped in "run"

If the test should only be run normally, then begin the name with an asterisk, e.g. *MyTest

Always set the random seed first if you do anything involving randomness. Note that includes some primitives you may not expect, such as ask on an agentset, crt, sprout, max-one-of, et al.

You can open a model at the start using OPEN>. see models/test for examples. By convention, we don't use OPEN> in other directories besides models/test (or extensions).

If the test uses a model that isn't used anywhere else (e.g. if it isn't part of the Models Library), it can go next to the test in the models/test directory.

Your extension should be in the extensions directory and have a tests.txt file in it containing command tests. see e.g. extensions/table/TableExtension.txt.

Extensions tests don't automatically run in NetLogo 3D if you just use ./sbt "te extension-name", but you can run them in 3D separately by doing env JAVA_OPTS="-Dorg.nlogo.is3d=true" ./sbt "te extension-name". Name suffixes like _2D and _3D are honored for extensions tests.

It's now possible to test NetLogo stack traces from both reporter and command tests. The easiest way to see how is by example. Here are examples of both:

*command-task-in-command-task-stack-trace
  O> run task [run task [print __boom]] => STACKTRACE boom!\
  error while observer running __BOOM\
    called by (command task from: procedure __EVALUATOR)\
    called by (command task from: procedure __EVALUATOR)\
    called by procedure __EVALUATOR

*Xor7
  __boom xor false => STACKTRACE boom!\
  error while observer running __BOOM\
    called by procedure __EVALUATOR

Notice the * at the start of the test name. this is to prevent it from running with the run command, since the resulting stack trace would be slightly different since the top level procedure is run instead of __EVALUATOR. It's OK that these tests don't run in run mode because we are only testing the stack traces, not the results.

To print values to standard out from within a test, use __stdout. For example,

O> crt 5
O> __stdout sort turtles

would print [(turtle 0) (turtle 1) (turtle 2) (turtle 3) (turtle 4)].