Testing

The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.

Verification - Checking against a specification - 'Have we built the system correctly?'

Confirmation by examination and through provision of objective evidence that specified requirements have been fulfilled.

Validation - tester becomes the advocate for the end-user - 'Have we built the right system?'

Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled.

Test objective

A reason or purpose for designing and executing a test.

Test object

The component or system to be tested.

Testing is not debugging

Testing - set of activities conducted to facilitate discovery and/or evaluation of properties of one or more test items. Debugging - The process of finding, analyzing and removing the causes of failures in software, executed by developers.

Quality assurance

Part of quality management focused on providing confidence that quality requirements will be fulfilled.

Quality

The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations.

Error (mistake)

A human action that produces an incorrect result.

Defect (bug, fault)

An imperfection or deficiency in a work product where it does not meet its requirements or specifications.

Failure

An event in which a component or system does not perform a required function within specified limits.

Root cause

A source of a defect such that if it is removed, the occurrence of the defect type is decreased or removed.

SEVEN TESTING PRINCIPLES

1. Testing shows the presence of defects, not their absence - testing can show that defects are present, but cannot prove that there are no defects.
2. Exhaustive testing is impossible - Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases.
3. Early testing saves time and money - To find defects early, both static and dynamic test activities should be started ASAP in the software development life cycle.
4. Defects cluster together - A small number of modules usually contains most of the defects
5. Beware of the pesticide paradox - if the same tests are repeated over and over again, eventually these tests no longer find any new defects.
6. Testing is context dependent - Testing is done differently in different contexts.
7. Absence-of-errors is a fallacy - it is a fallacy to expect that just finding and fixing a large number of defects will ensure the success of a system.

Coverage

The degree to which specified coverage items have been determined to have been exercised by a test suite expressed as a percentage.

Test basis

The body of knowledge used as the basis for test analysis and design.

Test activities and tasks

Test planning

Test analysis - The activity of establishing or updating a test plan.

Test plan - Documentation describing the test objectives to be achieved and the means and the schedule for achieving them, organized to coordinate testing activities.

Test monitoring and control - comparing actual progress against the plan and taking whatever actions are necessary to meet the mission and objectives of the project

Test monitoring - A test management activity that involves checking the status of testing activities, identifying any variances from the planned or expected status and reporting status to stakeholders.

Test control - A test management task that deals with developing and applying a set of corrective actions to get a test project on track when monitoring shows a deviation from what was planned

Test analysis

Test analysis - The activity that identifies test conditions by analyzing the test basis.

Test condition (charter) - An aspect of the test basis that is relevant in order to achieve specific test objectives. See also: exploratory testing.

Test design

Test design - The activity of deriving and specifying test cases from test conditions.

Test case - A set of preconditions, inputs, actions (where applicable), expected results and postconditions, developed based on test conditions.

Test data - Data created or selected to satisfy the execution preconditions and inputs to execute one or more test cases.

Test implementation

Test implementation - the activity that prepares the testware needed for test execution based on test analysis and design.

Test procedure - A sequence of test cases in execution order, and any associated actions that may be required to set up the initial preconditions and any wrap-up activities post execution.

Test suite (test case suite, test set) - A set of test cases or test procedures to be executed in a specific test cycle.

Test execution schedule - A schedule for the execution of test suites within a test cycle.

Test execution

Test execution - The process of running a test on the component or system under test, producing actual result(s).

Testware - Work products produced during the test process for use in planning, designing, executing, evaluating and reporting on testing.

Test completion

Test completion - The activity that makes test assets available for later use, leaves test environments in a satisfactory condition and communicates the result of testing to relevant stakeholders.

Differentiate the work products that support the test process

• Test planning work products - typically include one or more test plans. The test plan includes information about the test basis, to which the other test work products will be related via traceability information as well as exit criteria which will be used during test monitoring and control.

• Test monitoring and control work products - typically include various types of test reports, including test progress reports produced on an ongoing and/or a regular basis, and test summary reports produced at various completion milestones. All test reports should provide audience-relevant details about the test progress as of the date of the report, including summarizing the test execution results once those become available.

• Test analysis work products - include defined and prioritized test conditions, each of which is ideally bi-directionally traceable to the specific elements of the test basis it covers.

• Test design work products - include test cases and sets of test cases to exercise the test conditions defined in the test analysis. It is often a good practice to design high-level test cases, without concrete values for input data and expected results.

• Test implementation work products - include test procedures and the sequencing of those test procedures, test suites, and a test execution schedule.

• Test execution work products - include documentation of the status of individual test cases or test procedures, defect reports, and documentation about which test items, test objects, test tools, and testware were involved in the testing. Test completion work products - include test summary

• Test completion work products - include test summary reports, action items for improvement of subsequent projects or iterations, change requests or product backlog items, and finalized testware.