General

Causes of Failed Tests

We investigate a failed test from the most probable cause to least probable

• most probable
  • bug in new test input
  • bug in new test code
  • bug in new code
• probable
  • bug in old test input
  • bug in old test code
  • bug in old code
• improbable
  • bug in library
  • bug in Operating System
  • bug in firmware
  • bug in hardware

see also Test Driven Development

Notes

• Do not comment tests add @Disabled(junit5), @Ignore(junit4)

QA

1. What about testing the tests?

   https://softwareengineering.stackexchange.com/questions/11485/how-to-test-the-tests/11510#11510

Small Guide to JUnit

• https://drive.google.com/open?id=0B94RYjKLv-avSGMwTFVyR0JHekk

Tests

Type of tests

• UI - difficult due to lack of semantic inteface, you work with ui widgets, clicks, values
  • web
    • selenium IDE - Has drivers to connect to various browsers. You can trigger commands: go to url, read/enter fields, wait for some conditions.
      • working with java & selenium ide - https://www.youtube.com/watch?v=Re3Y6_3aVOE
      • https://medium.com/@katalon_studio/top-3-selenium-ide-alternatives-for-firefox-chrome-a3c4dd22d86b

Libraries

• assertj (facade fluent library to junit5)
• junit - the stable testing library
  • testng - a more dynamic library
  • junit3
    • the classical library
  • junit4
    • requires java5 (annotations)
    • https://stackoverflow.com/questions/6685730/the-differences-between-junit-3-and-junit-4
  • junit5
    • requires java8
    • https://junit.org/junit5/docs/current/user-guide/#overview-what-is-junit-5
    • architecture - https://developer.ibm.com/tutorials/j-introducing-junit5-part1-jupiter-api/
• mockito - to mock objects
• exception
  • java8 - http://blog.codeleak.pl/2015/04/junit-testing-exceptions-with-java-8.html
  • catch-exception to intercept exceptions
    • https://github.com/Codearte/catch-exception
    • https://github.com/Codearte/catch-exception/wiki/Installation
    • https://stackoverflow.com/questions/17772939/mockito-better-expected-exception-test-using-spy
• hamcrest - good matchers
• property based testing with generators and invariants
  • https://www.sitepoint.com/property-based-testing-with-vavr/
  • http://www.nurkiewicz.com/2014/09/property-based-testing-with-scalacheck.html
  • https://jqwik.net/property-based-testing.html
  • junit5 - alternative engine with PBT - https://jqwik.net/
  • vavr-test - small PBT from vavr
• mutation testing
  • http://pitest.org/ - Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived. To put it another way - PIT runs your unit tests against automatically modified versions of your application code.

Unit Tests

Good Tests

• Should
  • be self contained:
    • in order to pass shouldn't be preceded by manual/automatic execution of other tests/main/tools.
    • have with them all necessary resources (probably in src/test/resources)
  • generate temporary files inside target\ folder to follow Intro to Maven conventions. See Convention over Configuration
• Should not
  • have absolute paths - not portable between devs' environments
  • not contain complex code:
    • no flow control instructions
      • if - see junit's Theories
      • no try/catch

Integration Tests

• Should not
  • delete existing data from repositories (databases, servers)
  • execute code that is never executed in production
• Cleanup/Initialize before tests
  • The test must start from a known state (not necessary clean)
  • When test finish you want to manually inspect the state

Advanced

• Tipuri de teste
  • regression tests - expected is recorded and later used as reference, good for legacy systems and systems with complex behaviour, ideal for e2e
  • e2e - end to end tests
  • whitebox testing
    • internal testing: depends on the internals of the application, based on a lot more assumptions, fragile testing (tests are broken when the internals are changed), the internals are changed more than the contract
  • blackbox testing
    • test against the surface (api/interface) that is a published contract
  • fragile testing
  • mock testing
    • mock tests are too fragile because they are used usually in a whitebox testing context
    • are more behaviour tests
  • behavioural testing
• Testing bad practices
  • fragile tests
  • tests that are not working
  • degenerate tests - small ROI (Return Of Investment)
    • small tests that doesn't any value
  • 100% code coverage by tests
    • leads to testing the getters and setters
      • https://stackoverflow.com/questions/6197370/should-unit-tests-be-written-for-getter-and-setters
      • https://medium.com/@mabogert64/the-argument-to-test-your-getters-and-setters-b97a62c285e7
• bad tests lead to no test at all
• Who writes the tests?