By testing our deliverables we're making sure the we deliver quality to our customers. There are numerous ways of testing applications, but the most important thing is that tests should be(come) automated. An increasing pace in staying innovative requires to look into ways to delivering software faster without sacrificing its quality. With continuous delivery, a build pipeline automatically tests the software and deploy it to testing and production environments. Continuous delivery allows us to deliver new software any time of the day, sometimes even multiple times a day.

Manual testing multiple times a day, soon becomes impossible. Automating everything — from build to tests, deployment and infrastructure — is the only way forward.

This test pyramid describes the scope of test automation, where in parallel of the test automation, manual testing will be performed.

Automated tests consist of:

• Unit test
• API test
• UI test
• E2E test

For more information on test automation, please refer to the Nextens Test Automation policy.

Manual tests consist of:

• Functional test on US level (API, UI, XBRL and Reporting)
• E2E test on feature or epic level
• Exploratory test
• Acceptance test

The manual tests form the baseline for test automation. Although unit tests are created by the developers, these tests can be expanded or enriched by checking the completeness of the unit tests against the manual tests. Based on the scope and coverage of the unit tests, there needs to be decided which tests will be automated for API and UI. This will be a collaboration between developers and testers within the dev team. Ultimately the E2E test coverage will be determined.

The complete coverage is also based on the outcome of the risk assessment (priority) of the feature/US and product under test.

Testers should expand their control downwards (shift left) Testers have mainly been focusing on the manual test of the upper part of the pyramid: UI and E2E testing. While this seems highly effective since you're testing everything at once, there are some downsides: it is late, it is relatively slow, it is hard to test many different scenario's, and you need a full environment to run your tests on. 

It might make way more sense to test all kinds of scenario's using Unit tests and API tests, testing the fundamental components. These tests run a lot faster and can easily run during build time keeping the test feedback cycle a lot shorter.

Dev-Test collaboration The quality of the application is a joint effort between developers and testers. While testers "own" quality assurance, they don't always write all tests, especially if the testers don't have programming skills. Therefore the collaboration between testers and developers is becoming more and more important.

When deciding on test variaties it could be usefull to use the test quadrants, where:

• on the left hand side the focus is on verifying if the US/feature complies with requirements and acceptance criteria
• on the right hand side validating if the product is fit for purpose
• on the bottom level the focus is on technology
• on the top level the focus is on the business