Performance Comparison - pobrelkey/mockdemo GitHub Wiki
Performance shouldn't be a major consideration driving one's choice of mocking library; the overhead of a mock object library over any sane number of unit tests is minimal. Still, having noticed that performance had been raised as an issue in the discussion list of at least one other mocking library, I got curious as to how performance compares among the various mocking libraries available today. I cobbled together an informal benchmarking suite, results of which are below.
I wrote a simple class, Benchmocker, that repeatedly runs all the tests in the MockingLibraryComparison example project for each of six mocking libraries. Rather than run e.g. the Mockito test 1000 times in a row, followed by the EasyMock test, etc., the tests are run in cycles using a randomized order each cycle, to counter any performance gain/loss one mocking library may derive from running immediately after any other.
The test was set to repeatedly execute runs from 1 to 10,000 test cycles. The cases above 1,000 cycles should be seen as extreme stress tests - most real-world projects I've seen tend to break up their build into several segments that run in parallel before reaching this number of unit test classes over the life of a single JVM.
82 full sets of test runs were obtained (a scheduled reboot occurred in the middle of the 83rd set of runs). 80 of these sets of runs were analyzed - the first set was thrown out to counter possible startup effects (jars getting loaded into file cache etc.), while the last full and partial final sets were thrown out to counter possible increased system load ahead of the scheduled reboot which terminated the run.
Testing was done over a nine-hour period on a 3.33 GHz 12-core Xeon W3680 running Windows XP with 4 GB RAM. JDK used was Sun JDK 1.6.0_20. Tests were run serially, i.e. only utilizing one core. The machine was unattended and had minimal other system load during this time.
| library | average total run times (# test runs) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 5 | 10 | 25 | 50 | 100 | 250 | 500 | 1000 | 2500 | 5000 | 10000 | |
| !EasyMock 3.0 | 221.08 | 228.90 | 246.93 | 252.00 | 304.10 | 369.58 | 554.38 | 901.25 | 1698.15 | 3854.33 | 7500.64 | 14821.36 |
| JMock 1.2.0 | 32.40 | 37.60 | 43.08 | 61.95 | 89.03 | 135.85 | 266.60 | 509.10 | 1023.55 | 2638.08 | 5307.23 | 10839.00 |
| JMock 2.5.1 | 139.08 | 160.40 | 159.55 | 190.85 | 243.48 | 311.25 | 622.18 | 1129.18 | 2157.38 | 5253.56 | 10466.79 | 20803.69 |
| JMockit 0.999.10 | 64.88 | 104.05 | 155.20 | 303.88 | 544.25 | 1214.30 | 3522.93 | 7371.78 | 15586.53 | 41209.77 | 84129.59 | 170785.90 |
| Mockito 1.8.5 | 391.00 | 419.70 | 490.03 | 654.28 | 959.75 | 1634.15 | 3896.45 | 7616.45 | 15287.98 | 38894.18 | 78427.74 | 158137.74 |
| Moxie 0.9.1-SNAPSHOT (22 July 2011) | 70.25 | 87.23 | 99.73 | 136.35 | 208.08 | 345.25 | 746.78 | 1433.93 | 2918.70 | 7304.69 | 14661.15 | 29371.31 |
| library | time per test class (# test runs) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 5 | 10 | 25 | 50 | 100 | 250 | 500 | 1000 | 2500 | 5000 | 10000 | |
| !EasyMock 3.0 | 221.08 | 45.78 | 24.69 | 10.08 | 6.08 | 3.70 | 2.22 | 1.80 | 1.70 | 1.54 | 1.50 | 1.48 |
| JMock 1.2.0 | 32.40 | 7.52 | 4.31 | 2.48 | 1.78 | 1.36 | 1.07 | 1.02 | 1.02 | 1.06 | 1.06 | 1.08 |
| JMock 2.5.1 | 139.08 | 32.08 | 15.96 | 7.63 | 4.87 | 3.11 | 2.49 | 2.26 | 2.16 | 2.10 | 2.09 | 2.08 |
| JMockit 0.999.10 | 64.88 | 20.81 | 15.52 | 12.16 | 10.89 | 12.14 | 14.09 | 14.74 | 15.59 | 16.48 | 16.83 | 17.08 |
| Mockito 1.8.5 | 391.00 | 83.94 | 49.00 | 26.17 | 19.20 | 16.34 | 15.59 | 15.23 | 15.29 | 15.56 | 15.69 | 15.81 |
| Moxie 0.9.1-SNAPSHOT (22 July 2011) | 70.25 | 17.45 | 9.97 | 5.45 | 4.16 | 3.45 | 2.99 | 2.87 | 2.92 | 2.92 | 2.93 | 2.94 |
Mockito, despite being one of the best regarded of the major mocking libraries, is far and away the slowest, with JMockit following close behind. JMock 1 is fastest, but is antiquated and should be avoided. The remaining libraries perform broadly similarly. (Moxie is the slowest of these, but hasn't been the subject of any serious performance tuning - functionality and simplicity of syntax have been the focus of development to this point.)
That having been said, the difference in run times between fastest JMock 1 and slowest Mockito mocking libaries is at most 14.5 seconds over a large cycle of 1,000 unit test classes - in most real-world builds of that size, this is not enough of a time savings to justify migrating for performance reasons. Most projects looking to bring down their test run times will want to look at parallelizing their tests as a somewhat quicker win than migrating to a new mocking library - or should invest that time fixing performance bottlenecks in the application itself.
It should be noted that the tests used don't fully exercise the capabilities of the various mocking libraries - in particular the tests only use interface mocks and not concrete-class mocks. Further investigation into this area of performance is needed.