Software Testing

This course gives an introduction to systematic methods of testing and verification, covering a range of static and dynamic techniques and their use within the development process. The course emphasizes the view that design should be carried out with verification in mind to achieve overall project goals.

Learning Objectives

• Learn testing techniques and the situations in which they apply
• Apply real-world testing tools and frameworks
• Learn how to file bug reports
• Understand and apply different manual and automated software testing techniques
• Understand the importance of systematic testing

Topics

• General Software Engineering Concepts
• Black Box testing Techniques
• White Box testing Techniques
• Mutation testing Techniques
• Load testing Techniques
• Empirical Software Engineering
• Software Defect Prediction

General Software Engineering Concepts

Software Development Process

The Waterfall Method is BAD do not use it. Agile is good, use it.

All Software Verification Techniques

TL;DR test cases, execution path analysis, code review, formal proof.

• Testing: Exercising software to try and generate failures.
  • Pro: no false positives, Con: incomplete
• Static Verification: Identify all possible execution paths and potential problems associated with that path.
  • Pro: Complete, Con: false positives, expensive
• Inspection/Review/Walkthrough: Systematic group review of program text, i.e. code reviews, pair programming.
  • Pro: systematic, thorough, Cons: informal, subjective
• Formal Proof: proving that the program text implements the program specification.
  • Pro: strong guarantees, Con: complex, expensive.

Limitations of Software Testing

• Dijkstra: "Program Testing can be used to show the presence of defects, but never their absence".
• It is impossible to fully test a software system in a reasonable amount of time or money
• "When is testing complete? When you run out of time or money."

Levels of Software Testing

• Unit Testing: assess software with respect to implementation
• Module Testing: assess software with respect to detail design
• Integration Testing: assess software with respect to subsystem design (how it comes together)
• System Testing: assess software with respect to architecture design (Reliability, maintainability, usability)
• Acceptance Testing: assess software with respect to requirements (check against customer requirements)
• Regression Testing: ensure that new changes do not break the functionality of existing code

Black Box testing Techniques

Black box testing techniques involve looking only at the specification for testing inspiration. It involves careful consideration of only the input and output from the system/unit/method.

Boundary Value Testing

For all the details on Boundary Value Testing.

The BVT technique involves selecting (5) specific values from the input domain (min, min+, nom, max-, max). We then test the unit over the selected domain with the single fault assumption in mind; Failures are only rarely the result of the simultaneous occurrences of two (or more) faults. I.e. hold one value constant while cycling the rest. We often extend BVT in 5 ways: Robustness (include max+ and min-), Worst Case (ignore the single fault assumption), Robust Worst Case (combination of both), Special Value (use logical guess), and Random (randomly select values). Works well for independent variables, and physical quantities. Limitations include dependent variables, logical variables (e.g. a PIN), and boolean variables.

Equivalence Class Testing

For all the details on Equivalence Class Testing.

In ECT we partition the set of all test cases into mutually disjoint subsets whose union is the entire set. We then choose one test case from each subset. There are 4 ways to select test cases from these sets:

• Weak Normal: include only one test per subset in the input domain
• Strong Normal: include a test case for every subset
• Weak Robust: include a only one test case per subset both inside and outside the input domain
• Strong Robust: include a test case for every subset both inside and outside the input domain

ECT works well for independent variables, and physical quantities. Limitations include dependent variables, logical variables (e.g. a PIN), and boolean variables.

Decision Table Testing

For all the details on Decision Table Testing.

DT breaks up the unit into conditions and actions. The table describes what actions are to be taken with the conditions arise, we call these rules. Conditions represent input, Actions represent output. From this table we derive our test cases based on the rules.

The DT is limited by a complete set of rules that are consistent. Benefit is that it provides a plain english representation of the test cases.

White Box testing Techniques

White-box testing is based on specific knowledge of the source code to define the test cases and to examine outputs.

Path Testing

For all the details on Path Testing.

Path testing includes breaking the program text into a program graph whose nodes are statement fragments and edges represent control flow. From this graph we determine paths of execution, which turn into test cases. We define these with a grammar:

• Label: a node/statement fragment
• Loops: (nodes in the loop)*

Example: ABC(DEF)*DGL, the path DEF can be executed any number of times.

Limitations include concurrency issues. Works well for generating code coverage metrics.

Data Flow Testing

For all the details on Data Flow Testing.

Data Flow testing attempts to cut down on redundancy presented in testing all possible paths. To do so, we must segment out program text into 4 classifications:

• Definition: A node is declared for the first time
• P-Use: Predicate use (if, while)
• C-Use: Computation use (assigning another value to it)
• Kill Node: a return, halts execution

From this segmented code we create a program graph, labelling the nodes with the program segments as well as their classification. From the program graph we then break it up into execution paths with the following criteria:

• All-Defs: all the paths which define, or redefine a variable (from it's definition to a use).
• All-Uses: all reasonable paths that include the definition and all uses.
• All-P-Uses/Some-C-Uses: all paths which lead from a defining node to a P-use, if there is no P-use, then the path ends at the first C-Use
• All-C-Uses/Some-P-Uses: all paths which lead from a defining node to a C-use, if there is no C-use, then the path ends at the first P-Use

Benefits

Cuts down on the number of execution paths which need to be considered.

Limitations

Provides a lot of pre-test overhead (expensive).

Coverage Metrics

For all the details on Coverage Metrics.

This metric is often used in in industry, often we use coveralls on github. Our goal is to cover all possible branches of the code. By analyzing the execution paths we can determine the coverage metrics. Typical coverage target is 80-90%.

Benefits

Works well for providing a metric to analyze testing productivity.

Limitations

Coverage metrics are not always accurate.

Example

• May produce drastically different numbers
  • Assume two segments P and Q
  • P has one statement, Q has nine
  • Exercising only one of the segments will give 10% or 90% statement coverage
  • Segment coverage will be 50% in both cases

Mutation testing Techniques

For all the details on Mutation testing Techniques

The goal of mutation testing to to determine the adequacy of the test suite. Mutation testing attempts to manipulate your source code, run it through your test suite, to ensure that if the source code is changed that your tests behave accordingly. This is usually accomplished by a tool. If the tool returns with un-killed mutants than the test case is inadequate and should be redefined taking these into account.

Benefits

Effective way to measure the adequacy of test cases.

Limitations

High cost of generating the mutants and executing each test case against them.

Load testing Techniques

For all the details on Load testing Techniques

Load Testing attempts to mimic multiple users repeatedly performing the same tasks, take hours or even days. These tests show how the system reacts, if it can handle the requests. We use probabilistic models to determine how we place the load on our system.

Advanced Software Testing Topics

Empirical Software Engineering

For all the details on Empirical Software Engineering.

Empirical software engineering is a field of research that emphasize the use of empirical studies of all kinds to accumulate knowledge.

How it works, their application, underlying assumptions and limitations.

Software Defect Prediction

For all the details on Software Defect Prediction.

Software Defect Prediction (SDP) is the line of research that concerned with building prediction models, which leverage software metrics to predict defect-prone areas of a software. The motivation is predict where QA should spend more time testing/looking for bugs. Learning from past mistakes and help protect from future bugs. Metrics such as the amount of times a particular piece of code has been changed can effect if the system sees that portion of code as bug prone.

Bug Prediction Models

• Statistical: Naive Bayes, Multivariate Adaptive Regression splines model (MARS), Logistic regression, Linear regression.
• Tree-based: Decision treets, Random forests, Classification and regression trees model (CART), recursive partitioning, Support Vector Machine (SVM)