unittest - sc15000/python-testing-cookbook GitHub Wiki

Basic Use of unittest
Test Preparation and Cleanup
Increasing Test Output Verbosity
Specifying Which Tests to Run
Combining Tests from Multiple Test Case Classes into a Single Suite
Alternative test suite definition: Defining test suites elsewhere, within module-level functions
Specifying Test Suites on the Command-Line
Retrospectively Migrating Old Tests to unittest
Maximising Efficiency of Large Test Sets

Basic Use of unittest

We'll dive right in. In order to get the most commonly used Python testing framework running in its most basic form, we need to:

import the unittest module into our test script file, as this contains the test execution engine and the test case base class.
prefix each test method with test, thereby allowing unittest to automatically discover the test(s).
run the unittest executor (main())

Additionally, for readability, it is convention to:

name the test case class as for the DUT but appended with Test.

###Summary Example

import unittest # 1
...
DUTClass(object):
   ...
DUTClassTest(unittest.TestCase) # 4

   def test_high_values(self): # 2
      ...
   def test_low_values(self): # 2
      ...

if __name__ == "__main__"
   unittest.main() # 3

Full Example

See Basic unittest example

Key Notes

Provided are assertEquals, assertTrue, assertFalse, assertRaises.
Some are more useful than others. assertEquals, for example, will at least report the compared values on failure, whereas assertTrue will literally tell you that True != False, which is pretty pointless.
When comparing anything but Python's basic built-ins (lists, integers, strings, dictionaries), you'll need to define the object's __cmp__() method.

Test Preparation and Cleanup

unittest provides a means to configure the DUT and/or framework prior to running the test - the setUp() method - optionally followed by the complementary tearDown(). This allows us to put the system into a specific, known state prior to each test, as well as return it to its original state, allowing the next test to follow. If you find yourself leading and/or following several test methods with the same sequence of steps, this is a good indication that you'd benefit by abstracting it all away into a single setUp() and teardown() methods, as appropriate.

CAUTION: Don't be lulled into a false sense of security - any objects/data stores external to the test framework (i.e. outside of its control) are your responsibility to restore to their prior state following a test. This particularly applies to anything that has a history/memory, such as databases and file storage.

Summary Example


class DUTTest(unittest.TestCase):
   def __init__(self):
      ...
   def setUp(self):
      """
      Every test method in this class - i.e. each test - will follow a call to this method 
      """
      print "Preparing DUT for test"
      ...
   def tearDown(self):
      """
      Every test will be followed by a call to this method
      """
      ...
   def test_first_test_item(self):
      """
      Will be run after setUp(), and followed by tearDown()
      """
      ...
   def test_second_test_item(self):
      """
      Will be run after setUp(), and followed by tearDown()
      """
      ...
   ...

Full Example

See unittest example with setUp() and tearDown()

Key Notes

There is no setUp() or tearDown() method for any of the higher levels of the framework, such as test class, module etc., so every class and module must have its own setUp() and tearDown() methods defined. unittest2 provides this which, as a unittest backport of Python 2.7 and beyond new features, has an almost identical usage.
tearDown() lends well to integration testing, where external processes such as databases may have been opened or modified during the test and need rolling back and/or closing subsequently.

Varying Test Output Verbosity

Increasing the amount of feedback provided by unittest as the tests are executed is achieved through the use of unittest's TextTestRunner object:

Summary Example

...
if __name__ == "__main__":
   # First, create a suite of tests to pass to TextTestRunner
   suite = unittest.TestLoader().loadTestsFromTestCase(RomanNumeralConverterTest)
   unittest.TextTestRunner(verbosity=2).run(suite)

By specifying TextTestRunner's verbosity parameter, unittest will print the docstring of each test as its method is executed.

Full Example

See unittest example with increased output verbosity

Key Notes

The 3 available verbosity options are:

0 - quiet - prints no test progress
1 - default - prints a . for each successful test and an E or F for each that is either written in error or fails, respectively.
2 - verbose - ./E/F are replaced with the name of the test method being executed as well as the method's docstring

Verbosity can alternatively be specified from the command line via --quiet and --verbose options.

Specifying Which Tests to Run

unittest's TestSuite() class offers a means to select specific tests to run at the exclusion of all others. This is a particularly useful feature when focusing on a particular bug, which might only manifest in a subset of a Test Case Class's test methods.

At the command-line, we specify the tests to be run, which forms the command-line argument list.
The argument list is then iterated through within the code, where we add each successive named test to the test suite. 3. We then run the test suite as before.

Summary Example

   import sys
   if len(sys.argv) > 1:
      suite = unittest.TestSuite()
      for test in sys.argv[1:]
         suite.addTest(RomanNumeralConverter(test)) # We add a test, via its string name, from any Test Case class within scope
   else
      suite = unittest.TestLoader().loadTestsFromTestCase(RomanNumeralConverter)

   unittest.TextTestRunner().run(suite)

Full Example

See unittest example with specified tests

Key Notes

Notice that the key is in the fact that we can use TestCaseClass(test_name) to select a test method
Be sure to correctly spell the test method name when entering on the command-line; notice that our basic code does not first verify that the test method actually exists as an attribute of the Test Case class, so rather than simply skip the unknown test, the script raises an unhandled exception and aborts.

Combining Tests from Multiple Test Case Classes into a Single Suite

An obvious limitation of previous examples is that we could only select tests from a single test case class to run in a single test run/suite. Here, we present a simple way to combine tests from any number of Test Cases.

We do this simply by creating a suite for each Test Case class of interest and passing those suites as a list to the TestSuite() object:

Summary Example

   suite1 = unittest.TestLoader().loadTestsFromTestCase(RomanNumeralConverterBasicTest)
   suite2 = unittest.TestLoader().loadTestsFromTestCase(RomanNumeralConverterEdgeCasesTest)

   suite = unittest.TestSuite([suite1, suite2])

   unittest.TextTestRunner().run(suite)

Full Example

See unittest example with multiple, combined test suites

Key Notes

In practice, it is preferred for maintainability to define each test case class in a separate file, using another "test runner" script to import them and add their test methods to the suites.
As we shall see in the next section, besides uniting various types of tests for a DUT into a single test suite to run collectively, there is another very good reason to partition tests into different suites: as DUT tests increase in number, coverage and complexity, it becomes increasingly onerous to run a test suite containing the entire test case set. A suite that once took a few minutes may now extend into the hours. This acts as a deterrent from actually running the tests and not running the tests is no better than not having any tests at all. It is therefore better to partition the collection of tests into different suites - perhaps a quick, 10 minute, superficial suite, as well as other of increasing length and complexity, which may run into the hours and is better suited for overnight regression testing. This allows all tests to be run every day, whilst also allowing the developer to remain productive, running the quick 10 minute trests throughout the day to ensure nothing major has gone awry.

Alternative Test Suite Definition

It is possible to define the test suites elsewhere in the module, outside of both __main__ and the TestCase class. For example, we can define module-level functions that each compile a different set of TestCase tests into a suite and returns it to the caller. All that then remains is to call that function within __main__ to get the suite, and pass it to the TextTestRunner().run() executor.

Summary Example

def all():
    """All Test Cases in this module

        Compiling a suite list
    """
    suite1 = unittest.TestLoader().loadTestsFromTestCase(RomanNumeralConverterBasicTest)
    suite2 = unittest.TestLoader().loadTestsFromTestCase(RomanNumeralConverterEdgeCasesTest)
    return unittest.TestSuite([suite1, suite2])  

def basic_excluding_one():
    """All Test Cases in the Basic set, except the 'ones' test
       
        Using 'map' to pass each chosen test method by string to its Test Case class 
        constructor, which returns a TestCase object
    """
    return unittest.TestSuite(
            map(RomanNumeralConverterBasicTest, ["test_parsing_millenium", "test_parsing_century"]))

def empty_and_one():
    """Selecting just the invalid, empty-string test and the valid, ones test
     
        Individual test case addition
    """
    suite = unittest.TestSuite()
    suite.addTest(RomanNumeralConverterEdgeCasesTest("test_empty_roman_numeral"))
    suite.addTest(RomanNumeralConverterBasicTest("test_parsing_one"))
    return suite
...
if __name__ == "__main__":
     # Iterate through the list of TestSuite() generator functions
    for suite_func in [all, basic_excluding_one, empty_and_one]:
        print "Running suite '%s'" % suite_func.__name__
        suite = suite_func() # Calling the function returns a TestSuite() object
        unittest.TextTestRunner(verbosity=2).run(suite) # run the TestSuite() object, as per normal

Full Example

See Alternative Test Suite Definition

Key Notes

The example in this and those of the previous sections show the variety of methods available for compiling test suites:

Adding test methods explicitly, one-by-one
Combining test suites into a single suite via a list - i.e. TestSuite() objects can be passed to the TestSuite() constructor just as TestCase() test methods can.
Compiling test suites on the fly, vs creating a callable elsewhere in the module that will return a prepared TestSuite()
The convenience of being able to load all test methods from a TestCase() (TestLoader().loadTestFromTestCase())

Selecting a Test Suite from the command-Line

As varied as these methods are, they still all have the weakness that, embedded within the module somewhere is defined the selection of which suites to actually run. It would be better to provide a way to indicate which test suites to run from the command-line. Better still would be to present the user with a list of all test suites available, dynamically derived using Python's powerful introspection. The user could then select the desired suite accordingly. By using introspection, rather than hard-coding the list of suites available, as the DUT matures and more tests are written, the selection list will automatically update, provided the usual rules are folowed: a module-level suite definition function that returns a TestSuite() is defined, its name begins suite, and it has an accompanying docstring.

By using this technique, the person running the test needn't open any Python files and root around for suite definitions, which means the they need not be the person who actually wrote the tests - if the tester is unavailable one day, the designer can still run the tests in their absence. This data hiding/encapsulation and abstraction approach is a core principle of good programming practice.

Summary Example

...
def suite_basic_excluding_one():
    """All Test Cases in the Basic set, except the 'ones' test
    """
    return unittest.TestSuite(
            map(RomanNumeralConverterBasicTest, ["test_parsing_millenium", "test_parsing_century"]))
...
import sys # For accessing command-line arguments

def usage():
    """Dynamically discover all tests suites and present them to the user for selection"""
    print "No test suites were specified for running!"

    this_module = sys.modules[__name__]
    module_item_names = dir(this_module)

    suites = [getattr(this_module, suite) for suite in module_item_names if 
                    suite.startswith('suite') and 
                    callable(getattr(this_module, suite))
             ]
    # Display the list of available test suites, with their descriptions, to the user
    for suite in suites:
        print suite.__name__, " : ", suite.__doc__
...
if __name__ == "__main__":
    
    # If no test suite was specified at the command-line, the user needs some guidance:
    if len(sys.argv) != 2: # If just the script name was entered, or more than one suite/string was nominated
        usage()
        sys.exit()
    else:
        try:
            # Try to access the suite identified at the command-line
            suite = getattr(sys.modules[__name__], sys.argv[1])()
            unittest.TextTestRunner(verbosity=2).run(suite)
        except AttributeError:
            print "Unrecognised suite name '%s'" % sys.argv[1]

Full Example

See nominating a test suite from the command-line

Key Notes

It is left as an exercide for the reader to extend this example to accept multiple test suites specified on the command-line. It is also straightforward, by using the same introspection technique for TestCase() as we did for TEstSuites, to combine test case and test suite selection from the command-line. Better still would be to define a list of each in thei own script, directing the test runner to them from the command-line - this would avoid lengthy typing at the command-line as well as unsightly and potentially confusing/error-prone command-line commands.

Retrospectively Incorporating unittest into Test Code

In the event that test code is written independently of unittest, perhaps without the knowledge of its existence, it is straightforward to essentially wrap the test code in the unittest framework, bestowing it with all of unittest's usual benefits, anmely:

As a test framework, unittest understands that code is likely to fail - raising an unhandled exception as an indication of test failure cuts the test run in its tracks, thereby bypassing the remaining tests. Imagine a test suite of 100 tests where the first failed - that's 99 tests for which you have no idea of the DUT's performance. Expecting failures, unittest simply fails the tests when it detects an AssertionError exception, reports it as a test failure and gracefully moves onto the next. In the event of a syntax error, unittest reports the test as having errored but, again, moves gracefully on to the next test.
Python's assert is more intended as a safeguard - drawing attention to something unexpected - than to actually debugging and diagnosing: when an assert fails, Python just indicates the failing line of code. In contrast, unittest reports the actual comparison that caused the failure, providing a useful starting point.
unittest has a variety of assert* methods already provided, such as assertTrue, assertFalse, assertEquals etc.

The migration to unittest is as simple as wrapping each of the original test case methods in unittests FunctionTestCase() method. And that's it!

Summary Example

...
class RomanNumeralConverterTest(object):

    def __init__(self):
        self.converter = RomanNumeralConverter()

    def test_parsing_millenium(self):
        """Verify the DUT correctly parses millenia"""
        assert self.converter.convert_to_decimal("M") == 1000
...
import unittest

if __name__ == "__main__":

    tester = RomanNumeralConverterTest() # Create an instance of the TestCase class

    # Wrap each of the test methods in unittest's FunctionTestCase()
    unittest_tests = [
            tester.test_parsing_millenium,
            ...
            ]

    suite = unittest.TestSuite()
    for test in unittest_tests:
        testcase = unittest.FunctionTestCase(test) # This is the critical step - migrate old test format to unittest's
        suite.addTest(testcase) # Add the converted test to the suite
    
    unittest.TextTestRunner(verbosity=2).run(suite) # run the suite as normal

Full Example

See Retrospective unittest-fitting

Key Notes

Maximising Efficiency of Large Test Sets

Consider testing a mathematical function. For any such function there will be a set of inputs for which the function is designed to process and all else should be gracefully rejected. Furthermore, the function will expect either a collection (e.g. list) of input values, individual values, certain types of values and may or may not respond well to being passed None or empty collections. Clearly, there are many scenarios to verify, including either side of the upper and lower valid limits for data input values. Writing an explicit test for each of these conditions can be onerous. The task can be simplified by writing a generic 'verify' method, whose behaviour and stimuli values are determined by items in e.g. a list. For example, each test could be wrapped into a list entry, itself a list, comprising: [<expected result type>, <DUT action to exercise>, <input stimuli>, <expected output>]. By taking this approach, adding more tests in future is as quick and simple as adding an entry to the list.

Summary Example

   
class RomanNumeralConverterTest(unittest.TestCase):

    def setUp(self):
        self.converter = RomanNumeralConverter()
        self.rn_to_d = self.converter.convert_to_decimal
        self.d_to_rn = self.converter.convert_to_numeral

    def test_valid_input(self):
        """Verify that 'ordinary' valid input is correctly processed
        """
        # All we do here is define the test parameters
        tests = (
                    (self.rn_to_d, "equals", "CLXVIII", 168),
                    (self.d_to_rn, "equals", 19, "XIVIII"), # Deliberately failing test
                    (self.d_to_rn, "equals", 987, "DCCCCLXXXVII")
                )    
        # List comprehension is just a more succinct form of 'for' loop
        [self.verify(test) for test in tests]

    def test_corner_cases(self):
        """Verify that the DUT behaves correctly either side of the valid/invalid 
        value boundary
        """
        tests = (
                    (self.rn_to_d, "equals", "MM", 2000),
                    (self.rn_to_d, "raises", "MMI", ValueError),
                    (self.d_to_rn, "equals", 2000, "MM"),
                    (self.d_to_rn, "raises", 2001, ValueError)
                )
        [self.verify(test) for test in tests]
...
    def verify(self, test_params):
        """The workhorse of the TestCase test class - a generic approach to covering 
        all types of tests for this DUT class.
        """
        # Split for readability
        (test_method, test_condition, test_input, expected_test_output) = test_params
        
        # Provide test progress feedback
        print "\nTesting that '%s' (input) %s '%s' (output)..." % (test_input, test_condition, expected_test_output)

        # Test behaviour depends on condition:        
        if test_condition == "equals":
            self.assertEquals(test_method(test_input), expected_test_output)
        elif test_condition == "raises":
            self.assertRaises(expected_test_output, test_method, test_input)
        else:
            raise ValueError("Unknown test condition: '%s'!" % test_condition)
        
        print "PASS"


if __name__ == "__main__":
    suite = unittest.TestLoader().loadTestsFromTestCase(RomanNumeralConverterTest)
    unittest.TextTestRunner(verbosity=2).run(suite)

Full Example

See Testing by iteration

Key Notes

Note that what we gain in scripting convenience, we lose in test feedback, so this is arguably a retrograde step. Since unittest is not aware of multiple 'tests' within a single test method, it will abort on the first failure. See the example above, where the failure of the "deliberately failing" test causes subsequent test steps within that same test method to be aborted. It is recommended to use this approach for algorithmic DUT operations, which simple churn through data sets, as opposed to mechanically different tests, which may access external databases, processes etc.

unittest - sc15000/python-testing-cookbook GitHub Wiki

Contents

Basic Use of unittest

Full Example

Key Notes

Test Preparation and Cleanup

Summary Example

Full Example

Key Notes

Varying Test Output Verbosity

Summary Example

Full Example

Key Notes

Specifying Which Tests to Run

Summary Example

Full Example

Key Notes

Combining Tests from Multiple Test Case Classes into a Single Suite

Summary Example

Full Example

Key Notes

Alternative Test Suite Definition

Summary Example

Full Example

Key Notes

Selecting a Test Suite from the command-Line

Summary Example

Full Example

Key Notes

Retrospectively Incorporating unittest into Test Code

Summary Example

Full Example

Key Notes

Maximising Efficiency of Large Test Sets

Summary Example

Full Example

Key Notes