Writing a Regression Test - ngnjs/chassis-lib GitHub Wiki

Why?

For any number of reasons, it can be hard to recreate an issue experienced by someone else. If nothing else, the process can sometimes be time consuming. Using regression tests can ease this problem, since they can illustrate the problem.

How To

First, create an issue explaining the problem. Take note of the issue number. Next, fork the repository and prepare to modify the regression tests. Our regression tests exist in regression.js, where you can see some examples of existing tests.

The idea is fairly simple: create a test that should pass, but actually fails. It is then up to us to make the test pass (i.e. fix the issue).

For example, let's say we have the following erroneous source code:

window.add = function (a, b) {
  return a + b + 1
}

You would expect add(1, 1) to return 2, but it actually returns 3. This is obviously a problem. The regression test (in regression.js) might look like:

test('Regression', function (t) {
  /**
    * @issue 123
    * Incorrect addition.
    */
  t.ok(add(1, 1) === 2, 'Adding two numbers generates a proper sum.')
  t.end()
})

The test above basically says adding 1+1 should equal 2. That's what the code should do, but currently doesn't. When we run this test (or when it runs on Travis CI), it will not pass. When you submit this as a PR, this will get our attention. To distinguish between a normal PR and a regression PR, please leave a comment on your PR (remember it will fail to pass CI tests, which is OK). It gives us an example of what the expectation is, where to look in the code, etc. We would then fix the offending code, i.e.

window.add = function (a, b) {
  return a + b // Removed + 1
}

Once the test passes, we'll push an update.

NOTICE: The regression test should contain a reference to the issue in the comments right above the actual test. Our documentation parsers look for @issue, effectively allowing us to keep an audit log of changes. We can automatically update the documentation this way.