2 Designing Tests And Test Automation Strategies - essenius/AcceptanceTesting GitHub Wiki
Knowing technically how to create and run automated test cases is important, but not enough. You also need to ensure that your test suites are effective (uncover as many defects as possible) and efficient (use as few test cases as possible). I have learned a lot from the Construx training “Professional Tester Boot Camp”, and many of the topics described in this chapter will look familiar to people who followed that course.
Let us start with a fact of life in testing: It is not economically possible to do exhaustive testing. Testing can prove the existence of defects, but not their absence. At best, it can provide a “suitably convincing demonstration” that any remaining defects are either unlikely to be present, or of little consequence (seriousness) if even if they were present. The techniques on this page describe what options are available to make this demonstration convincing enough, and to minimize the risk of missing a crucial factor.
The page focuses on functional (black box) testing since that is also the sweet spot of FitNesse. Structural (glass box) testing is not covered as that is the domain of unit and unit integration testing, which is typically not done with FitNesse.
What is a Defect?
A defect is an inconsistency between system’s implementation and either the user’s expectation or the specification. This definition is quite broad, and purposely so. Coding defects are very clear: the implementation in code is not corresponding to the design. These can be things like using an assignment operator (= in C# or Java) where a comparison (==) was intended, or not adhering to the coding standards. Design defects can be for example errors in the logic, issues in error handling and recovery, problems in inter-process communication, high class coupling (fan-out), high cyclomatic complexity of methods, or testability issues.
But projects tend to make more than half of their mistakes in the requirements [Requirements Based Testing: An Overview by Gary Mogyorodi]. And those mistakes are not only the most frequent ones, they are also the most expensive to correct. Mogyorodi claims that 4 out of 5 dollars spent on defects are used to correct requirements defects. But what precisely is a requirements defect? In essence, it is a discrepancy between what the user wants an application to do and what it actually does.
For example, if the requirements elicitation process has been done poorly and the specification states a wrong requirement, that is a requirements defect. Many developers will then claim that it is not a defect, since the product works according to the specification. But even though the developer may have implemented the code exactly according to the specification, the code doesn’t do what the user expected it to do, so there is definitely a defect. In this example, the specification itself was defective. Another example of a defect is an incomplete specification. Developers then (often unknowingly) make judgment calls to fill in the blanks, which can very well be wrong because the developer doesn’t necessarily have the business context to oversee the consequences of these decisions.
There is of course a fine line here. We have a defect if something that the user expected all along hasn’t been delivered. But also, the users will evolve their thinking over time, and may change their minds on what they think the system should do. In those cases, if the system functions according to the earlier expectation, then there is no defect, even though the user now expects something else.
How Can We Detect Defects?
Test cases are intended to reveal defects. If a test case fails, there is a defect somewhere (possibly even in the test case itself). But testing is not the only way in which defects can be detected. Another very powerful strategy is using inspections (popularized by Michael Fagan in the 1980’s). The advantage of applying inspections is that this can be done on any deliverable: project plans, requirements, designs, test plans, user documentation, and so on. You need code for testing, but you don’t need code for inspections.
There is another reason why inspections are important. As stated above, the most often occurring types of defect are requirements defects. To make things worse, a defect gets more expensive to correct as it stays in the system longer (see e.g. “What We Have Learned About Fighting Defects” by Shull et al.), or more accurately, it gets more expensive as more decisions have been made based on the defect. If you make a mistake in the requirements, you can design the system wrongly, construct it wrongly, and only find out during testing that the requirement was wrong. This means a whole lot of rework: correct the specification, correct the design, correct the code, and re-test to ensure the corrections were done right. So, it makes sense to try and find these requirements defects early. Inspections allow you to do just that. Requirements defects identified in inspections are usually quite easy to fix: just correct the specification.
Applying Acceptance Test Driven Development (ATDD) will provide you a kind of instant inspection: in the process of creating test cases, there will be a dialog between product owner, developer and tester to iron out ambiguities (which are defects waiting to happen).
Boris Beizer stated in his book “Software Testing Techniques”:
More than the act of testing, the act of designing tests is one of the best bug preventers known.
This is exactly the reason why ATDD works. It is much better to design test cases upfront than after development, because it will prevent many defects from being injected. This in turn will cause the amount of unanticipated rework in a project to be substantially less. ATDD expresses requirements as unambiguous test cases instead of ambiguous natural language. Supplementing ATDD with coverage techniques as described in this chapter can also help alleviate the incompleteness problem.
The bottom line is that you should never rely solely on testing to discover defects. Testing is an important part of the quality strategy, but it should not be your only way to find defects. The mindset is the most important aspect: there should be zero tolerance for defects, and defects should be identified and eliminated as soon as practically possible.
What is a Good Test Case?
A test case is a specific set of input values given to an element of software in a certain state with intent to expose a certain kind of defect. A useful way to structure test cases is Arrange - Act - Assert:
- Arrange: prepare the input values and system state
- Act: execute the function being tested
- Assert: check whether the actual results are corresponding to the expectation.
Good test cases are simple and fast to execute, are repeatable, make failure obvious, distinguish various kinds of failures and have a reasonable chance of exposing previously unknown defects.
Positive test cases check whether a piece of software does what it is supposed to do. However, that is just half the story. The second half is to ensure that the application does not do what it is not supposed to do. And that is what the negative test cases are intended for. People have a natural tendency to focus on positive test cases, thus leaving a lot of potential for defects caused by lack of negative test cases to go undetected. So, it is important to give negative test cases sufficient attention.
For example, consider testing a cash withdrawal application. A positive test case would be to withdraw $100 from an open account that has a balance of $1,000: this should succeed. Negative test cases would be to try and withdraw $100 from a closed account and from an account with a balance of $80 which does not allow an overdraft. Both should not result in a withdrawal.
Notice that you can overload positive test cases, but not negative ones. A single positive test case can test for as many positives as practical. Negative test cases can test only one negative: you must be very specific as to what it is that you don’t want the system to do. For example, suppose that you test a password strength validation which has two rules: the password must have at least 8 characters, and it must be using a combination of alphanumeric characters and numbers. Then you can create a single positive test case that meets both rules. But you will need to create separate test cases to validate that failure to comply with each rule is detected, i.e. whether passwords with less than 8 characters are rejected, and whether passwords with 8 or more only alphanumerical characters are rejected.
A key ingredient of a good test strategy is making sure to select the right test values. How to do that is discussed in the section Domain Coverage.