See also this blog post

See screenshots at the bottom of this page

SoarUnit is a framework for unit testing Soar code introduced in JSoar 0.10.1. It is implemented as part of JSoar, but it supports running code in either JSoar or CSoar 9.3.1. When run, SoarUnit scans a given directory for test cases (see below) and runs all of the tests that are found. It then reports the test results, either in a text format, or through a user interface similar to the JUnit view in Eclipse.

SoarUnit was created to support development of the Bebot Soar library. There are numerous test examples there. Note that the tests are only compatible with JSoar.

Example tests compatible with both JSoar and CSoar (for the waterjug demo) are available here.

The run loop of SoarUnit looks like this:

• Create a new agent and cd to the directory containing the file of the current test
• Source code in the setup block
• Source code in the current test block
• Run the agent and wait for the (pass) or (fail) RHS functions to be called (see below)
• Record test results
• Destroy the agent
• Repeat

Currently a test case is identified by SoarUnit as a file that starts with "test" and ends with ".soar".

A SoarUnit test case consists of a setup block and one or more tests. The setup block can source any code needed to setup up the tests. As noted above, the setup block is executed before each test is run. Here's an example of a setup block:

setup {
  source ../common.soar

  sp {apply*init*add-state-name
    (state <s> ^operator.name init)
  -->
    (<s> ^name test-list-get)
  } 

  ...

Individual tests are specified with test blocks. Each test block has a name and a set of test-specific code that implements the test:

test "Test multiply multiplies value by factor" {
  sp {propose*identity
    (state <s> ^name test-functions
              -^result)
  -->
    (<s> ^operator <o>)
    (<o> ^name bebot*multiply
         ^value 99
         ^factor 3)
  }

  sp {pass
    (state <s> ^name test-functions
               ^result 297)
  -->
    (pass)
  }
} 

The rules in a test (and setup) have access to two new right-hand-side functions, (pass), and (fail). Both functions cause the test to halt. (pass) indicates the test has passed. There should always be a rule to test for success and call (pass). On the other hand (fail) can be called by tests that detect a failure condition in the test. Both functions can take arbitrary arguments which are used to construct a message which is included in the test report.

While running tests, SoarUnit keeps track of loaded rules and the firing counts of those rules. This information is used to perform a rudimentary code coverage calculation. Any rule that isn't fired counts against code coverage.

SoarUnit is a command-line application which is run with bin\soarunit.bat on Windows and bin\soarunit on Unix systems. The simplest way to run SoarUnit is to point it at a directory:

$ soarunit /path/to/tests

this will run all tests found in the given directory. To run SoarUnit with CSoar rather than JSoar, set the SOAR_HOME environment variable and use the --sml switch:

$ export SOAR_HOME=/path/to/csoar-9.3.1
$ soarunit --sml /path/to/tests

Use the --help option for all command-line options.

Known Issue: --sml only seems to work if -R is also specified.

SoarUnit has a graphical user interface which displays the familiar green bar from of other unit testing UIs. To run the run use the --ui option:

$ soarunit --ui /path/to/tests

Here's the UI with all tests passing:

Right-clicking a test gives options for editing and debugging a test.

Here's the UI with failing tests:

/soarunit//soarunit-fail.png

Here's the code coverage user interface:

/soarunit//soarunit-fail.png

Your actual system may have RHS functions in it that your agent uses. You want to test the agent in SoarUnit, but the RHS functions are not available there. What to do?

First, consider whether your RHS functions could be built into the Soar code (e.g., using Tcl, jython, jruby, or javascript). If not, then we can use Tcl to replace all the calls to the RHS functions with calls to Tcl stubs. Here's an example:

set soarUnitRhsNames "myRhsFunc yourRhsFunc"

rename sp spSoarUnitInternal

# this version of sp replaces all calls to identified RHS functions with calls to the tcl versions
# e.g., (myRhsFunc <foo>) becomes (tcl |myRhsFunc <foo>|)
# to simplify, we will assume that the next closing paren after the RHS function name is the end of the call
# i.e., we will not look for escaped or nested parens
proc sp { body } {
    global soarUnitRhsNames
    foreach rhsName $soarUnitRhsNames {
        # find all instances of myRhsFunc, capture starting at myRhsFunc up to first closing paren, and replace with (tcl |captured stuff|)
        # \((myRhsFunc [^\)]*)\) = find an open paren followed by myRhsFunc, and then any non-closing paren chars followed by a closing paren char
        #  capture the part between the open and close parents
        # (tcl |\1|) = replace with a call to the tcl RHS func with the original call passed as a string arg in verical pipes
    
        # we want to substitute the variable name, but all the square brackets and backslashes in a regex break normal substitution
        # so we do a special substitution that ignores backslashes and square brackets to avoid having to escape everything
        set regex [subst -nobackslashes -nocommands {\(($rhsName [^\)]*)\)}]
        
        set body [regsub -all $regex $body {(tcl |\1|)}]

        # the way Soar variables are substituted is different, so we need to adjust for that
        # in a normal RHS function call, something like (myfunc <foo>) will substitute the bound value for <foo>
        # but when it's converted to a tcl call, it looks like (tcl |myfunc <foo>|) -- now the variable is just embedded in a string and won't be substituted
        # what we want is actually (tcl |myfunc |<foo>), or actually (tcl |myfunc |<foo>||) since the variable may not be at the end of the command
        # note we can't just change the original call to put pipes around the variable, as this would work for tcl but not the original (actual) function call
        # so what we'll do is find all instances of tcl $rhsName and replace any <*> with |<*>|
        # this is a little aggressive, but will work in the vast majority of cases
        # (\(tcl \|cs2tts )(\<\S+\>)([^\)]\)) = find an open paren followed by tcl MyRhsFunc space, then a Soar variable, and then any non-closing paren chars followed by a closing paren char
        #                                       capture the Soar variable and the stuff before and after
        # \1|\2|\3 = replace the Soar variable with the Soar variable in pipes and leave the rest as it was
        set regex [subst -nobackslashes -nocommands {(\(tcl \|$rhsName )(\<\S+\>)([^\)]\))}]

        set body [regsub -all $regex $body {\1|\2|\3}]
    }
    
    spSoarUnitInternal "$body"
}

proc myRhsFunc { args } {
    return whatever
}

proc yourRhsFunc { args } {
    return something
}

What this essentially does is intercept all calls to sp, and then pre-process the strings that make up the rule bodies to replace any calls to identified RHS functions with calls to tcl procs of the same name. You can then write those Tcl procs to do whatever you want (since this is just testing, returning hard-coded values is often sufficient). A limitation of this is that calls to the tcl RHS function always return strings, so if you want to return numbers (or even structures), you might want to modify this to add the appropriate casts for each RHS function. It may also be possible to use one of the other scripting languages instead (I think you still need to use Tcl to intercept the sp command, but you may be able to substitute a call to, e.g., the javascript RHS function instead of tcl.