Introduction

It has become an issue that our tests take so long - it slows down development and especially the release process where every small change needs ~1 hour of wait time while tests run. We should be able to run tests really quickly and be agile in our development. In this page I present an analysis of things that cause friction in the product release cycle, with proposed improvements. Things have got to the point where often we are spending more time maintaining tests than writing new features and fixing bugs. So lets fix that!

Slow tests

There are a number of reasons why tests are slow including:

• Tests often overlap in functionality - each test should add coverage and not duplicate other tests coverage more than necessary
• Tests for dead code - we have quite a lot of dead code (that is code that is not actually used in production) -if we remove dead code and the tests related to it we can reduce test time.
• Tests using overly large datasets - testing with large datasets prolongs test time because analysis is more time consuming
• Tests without mocks - in some cases we use expensive setup / destroy procedures for objects (e.g. canvas, registry) - we could be using a mock object in places to simulate the backend objects that are not central to the test

Proposed actions:

• Vigorously remove all non-production code from our code tree
• Judiciously remove tests that do not add to code coverage
• Create small tests datasets for tests and do not run tests for regions / national datasets
• Evaluate which classes can be mocked (e.g. QgsMapCanvas) and use mocks instead

Render checking tests

• Testing is not only physically slow (in wait times) but also certain types of tests are slow to maintain. In particular render checker tests vary so much between hosts, hosts, system architectures and software versions that they can no longer be considered a reliable way to determine rendering fidelity.
• Render tests need human verification of anomalies which is not possible when using e.g. Travis as testing service since we do not have physical access to the file system in order to verify them.

Proposed actions

• Remove all render checking tests
• Try to come up with alternative novel ways of validating that pdf and raster outputs are being generated correctly

Test data in a separate repo

• Our test datasets are stored in a separate repo and are huge. If you do a 'clean state' test as e.g. travis does, a full checkout of the test datastore is required which is time consuming.
• If we use a local persistent test storage (as we did in Jenkins) it wastes a large amount of space for each test data checkout.
• If we use a shared persistent test storage for multiple test jobs (as we did on jenkins) there are potential race conditions and tests need to be run in series and there is a chance that unwanted state changes in the test data repository occur.

Proposed actions

• Immediately discontinue use of inasafe_data repository for unit tests (the repo is still useful for integration tests)
• Refactor all tests to use small geographic areas and small cell counts (for rasters) / feature counts (for vectors). This will have an added benefit that application logic is easier to validate manually.
• Reduce the geographic variation of tests. For each layer we have in our standard layers dataset there is an over head in loading / unloading the layer for each test run. By using the same layers for more tests we can reduce setup/teardown overhead.
• No test data file should exceed 100kb. This is a rule of thumb to keep our repository size small.
• No test dataset should include more than 1000 features. This is a rule of thumb to keep our test times low and file sizes down.
• We should use simplified boundaries for e.g. aggregation tests to keep computation times low. e.g. using simple squares instead of kabupaten will speed up feature selections and also make test validation simpler.
• We should simplify all data layers to remove pseudo nodes.
• We should have a restricted number of layers, one for each hazard and exposure type supported and one aggregation layer for each geographical test area.
• All lic files and checksums need to be updated based on the reduced datasets.
• All tests that use other layers than those provided in our standard layers need to be updated to use them.
• All tests should have their output scores validated against the new datasets and then the test characterisation updated accordingly (e.g. expected affected people changed from XXX to YYY)
• As a coding philosophy point I also believe that it is better for the code and the test data to be in one place - new developers should be able to immediately check out the code and run tests without doing any additional hoop jumping.

Strong linkage between project documentation and application documentation

• Currently the context help of InaSAFE is sourced by a rather complex process of extracting a subset of the global project documentation, copying it over to the code source tree and replacing any prior documentation files there. All image resources are then replaced and we get a git changeset that numbers around 385 files (even if only a few files were actually altered).
• Changing application help documentation in the global project documentation causes flux in the documentation team as they have to redo screenshots etc. to match the global documentation project.

Proposed actions

• Remove the application help sources from the inasafe-doc repo completely and maintain those documents in source for easy translation via transifex.
• Use our html generator logic (as done in e.g. the dialog text of OSM downloader dialog) to generate the actual html sources.
• Store images as local resources in the code tree.

Strong linkage between project translations and application translations

• Currently if we want to change contextual help or dialog text in the application, we break the greater documentation teams workflows.
• We need to have the flexibility to apply last minute tweaks and improvements to application text with minimal flux.

Proposed actions

• Separate the application translation work to its own transifex project and manage those translations in-house (to the developer team).

Repository size

timlinux@overhang:~/dev/python$ du -sh /tmp/inasafe/
183M	/tmp/inasafe/

• Our repository has grown large - doing a checkout of the repository takes longer than it needs to on a fresh machine (e.g. travis) - as you can see from the above code listing, a clean checkout is ~183mb.
• We have a lot of unused code (as mentioned above) and tests for them, history of control images and other bloat in our code base.

Proposed solution

• After doing the refactoring outlined in this document, create a new repository (e.g. http://github.com/AIFDR/inasafe3) which contains and export / import of the code base (thus truncating all history.
• Rename all new releases of InaSAFE from version 2.x to 3.x and base them on the new code base

Goals

• Test should run in under 10 mins on travis
• Repository should be under 10mb for a clean checkout (ok it will grow again over time)
• No unused code / test IF's / deprecated IF's
• Application documentation and translations workflow is independent of project documentation / translations
• We have a clean platform on which to start our 'use QGIS to the max' effort