Reporting and Analyzing Bugs

Failure, fault, error and incident

Failure

• Observable incorrect behavior of a program. Conceptually relate to the behavior of the program, rather than its code
  • A failure occurs when a fault executes

Fault (bug)

• Related to the code. Necessary (not sufficient) condition for the occurrence of a failure
  • A fault is the result of an error
  • A fault won’t yield a failure without the conditions that trigger it.
  • Example: if the program yields 2+2=5 on the 10th time you use it, you won’t see the error before or after the 10th use.

Error

• Cause of a fault. Usually a human error (conceptual, typo, etc.)
  • “People make errors”

Incident

• The symptom associated with a failure that alerts the user to the occurrence of a failure

Types of Errors

It is common practise to report all the following types of problems, but remember to specify the type of error you are reporting on the bug report.

• Coding Error: program doesn't do what the programmer whats it to.
• Design Issue: Intended behaviour, but confusing.
• Requirements Issue: Intended behaviour, easy to understand, but fails to meet a specified requirement.
• Documentation/Code Mismatch: Report to programmer and writer.
• Specification/Code Mismatch: method specification doesn't match the code or vice versa.

Bug Reporting

To write a fully effective bug/feature report you must:

• Explain how to reproduce the problem.
• Analyze the problem so that it can be described with a minimum number of steps.
• Write a report that is complete, easy to understand, and non-antagonistic.
• Remember you're trying to sell the programmer on fixing your bug (motivate and overcome objections).
• Investigate, sometimes a failed test can mean many things.
• Write follow-up tests.
• Is the problem new to this release?
• Always report non-reproducible errors, the programmer may be able to figure out the underlying failure (race conditions, data loss, etc.)

Example Problem Report Form

A typical form includes many of the following fields

From QA

1. Problem Report Number
2. Reported By
3. Date Reported
4. Program (or component) name
5. Release Number
6. Version (build) identifier
7. Configurations
8. Report Type
9. Can Reproduce
10. Severity
11. Priority
12. Problem Summary
13. Key words
14. Problem description and how to reproduce it
15. Suggested fix
16. Status

From QA Manager
17. Resolution (Pending, Fixed, Cannot reproduce, Deferred, As designed, need info, duplicate, withdrawn)
18. Resolution version
19. Resolved by
20. Resolution tested by
21. Change history
22. Comments

Testing Oracles

• Consistent with History
• Consistent with our Image
• Consistent with Comparable Products
• Consistent with Claims
• Consistent with User's Expectations
• Consistent with Product
• Consistent with Purpose