Testing Pattern catalogue

The test pattern catalogue defines a set of recurring concepts, terms, and approaches which arise in the field of regulatory testing. By giving names to these it makes it easier to discuss them and to collaboratively define good standards for tests and testing tools.

Test Definition

A definition of a test, for a Test Scenario, It does not need to include Test Input Data

Test

A Test definition plus Test Input Data, and optionally Expected Results

Executed Test

A Test which has been run using Executable Regulatory Logic and has some as results

Regulatory text

Some text which is part of the regulation which describes Regulatory Logic

Regulatory Report

A Report which includes many Regulatory Report Cells and is often the end result of Regulatory Logic .

One-Dimensional Regulatory Report

A Regulatory Report which is a set of field names, and may have mutliple repeating rows.

Two-Dimensional Regulatory Report

A Regulatory Report which is split into set of rows and column with fixed meaning

N-Dimensional Report

A Regulatory Report where there may be more dimensions, such as multiple sheets following the same 2 Dimensional Pattern

Regulatory Report Cell

A cell in a Regulatory Report

Test Input Data

Input data for a Test

Structured Input Data

Test Input Data which follows a specific structure

Tabular Input Data

Test Input Data which follows a tabular structure, where tables have named columns and columns may have types

Structured File Input Data

Test Input Data which has structured fie as input. The structured file could have a tree structure such as structured XML documents that follow an XML Schema Definition

Test Output Data

The Resulting Data from running Test

Test Output Data Structure

A definition of the structure of Test Output Data

Structured Test Output Data

Test Output Data which follows a specific structure

Report Cell Based Test Output Data

Test Output Data which is for a specific Report Cell, or set of Report Cells (like a full Report or set of Reports

Tabular Test Output Data

Test Output Data which is for a specific Tabular DataSet Structure

Input Data Model

A Structure for Test Input Data, such as a relational schema or XML Schema Definition

Standard Data Model

An Input Data Model which is provided for use with a regulation, by a regulatory body.

Harmonised Input Data Model

An Input Data Model which aims to provide input data form multiple regulatory reports perhaps across multiple regulations

Specific Input Data Model

An Input Data Model which focusses on one, or a small number of reports in one regulation.

Expected Results

Test Output Data that we expect from a Test

Test Scenario

A particular Scenario from Regulatory Logic that we want to test

Test Scenario Taxonomy

A way of organising Test Scenarios in order to reflect some level of completeness.

Test Scenario Grid

A way of organising Test Scenarios as a grid or n dimensional cube, for example with asset class (e.g. bonds, loans, derivatives) and risk type (e.g. credit risk, market risk, liquidity risk) as the 2 dimensions.

Sparse Scenario Taxonomy

A Test Scenario Grid where there are many grid co-ordinates that are empty (for example if the Regulatory Logic means we are not required to calculate a particular risk type for a particular asset class). If it is unclear which co-ordinates should be empty then this can affect a good measure Coverage.

Coverage

Good Coverage measn that a set of Tests covers a large proportion of Test Scenarios, and that the Test Scenarios cover a good set of the possible Test Scenarios of the Regulatory Logic

Test Boundary

Test Boundary is a definition of what is our input data model and output data structure

End To End Test Boundary

A Test Boundary where the Input Data Model is the ultimate Input Data Model, not an intermediate DataModel, and the output data structure is the ultimate output data structure (such as report cells)

Unit Test Boundary

A Test Boundary which may be used to test an internal component of Executable Regulatory Logic, this might be implementation specific if there is not Standard Executable Regulatory Logic adnd is more likely to be a Technical Test It might change more often than an End To End Test Boundary

Technical Test

A test which uses concepts not familiar to regulatory domain experts

Business Test

A test which is focussed on concepts familiar to regulatory domain experts

Regression Test

A test which is used to highlight if\where new changes to Input Data Model, Output Data Structures and Executable Regulatory Logic have changed results of processing unexpectedly from a previous version. In some cases expected results of a regression test can be defined as the actual output of running the test on a current version of the Executable Regulatory Logic

Domain Test

A Domain Test has expected results which were calculated independently by regulatory domain expert, potentially with a Worked Example

Continuous Integration

A means of running a set of tests after changes to Input Data Model,Output Data Structures and Executable Regulatory Logic

Regulatory Logic

The logic of a regulation as defined by the regulatory text, this might define in natural language (like English) calculations, workflow of processes, and conditional processing.

Executable Regulatory Logic

This is logic of regulation defined in a way that computers can execute it

Standard Executable Regulatory Logic

A set of Executable Regulatory Logic provided by a regulatory body for a regulation.

Multi-Layer Regulatory Logic

This is Regulatory Logic which has multiple Layers of processing, such as netting (e.g securities netting) followed by enrichment (e.g. calculation of exposure value) followed by matching (e.g. matching exposures and collaterols)

Multi Step Regulatory Logic

This is regulatory logic where the calculation of a result has multiple steps.

Aggregating Regulatory Logic

This is regulatory logic where aggregation of multiple inputs is required to calculated results (e.g. netting of loans), so results cannot be calculated for each individual input separately (e.g. a calculation done separately for each loan)

Transactional Reporting

Regulatory Logic for reporting transactions, often this is not aggregation regulatory logic, and calculations can be done separately per transaction, it may be multi-step

Statistical Reporting

Some statistical reporting is Single Step as opposed to multi-step regulatory logic (i.e. there is not much calculation in between the Input Data and the output data, although often it will be Aggregating Regulatory Logic.

Prudential Regulatory Logic

Prudential regulatory logic is often Multi-Layer Regulatory Logic , Multi Step Regulatory Logic, and Aggregating Regulatory Logic

Narrow Input Data

If a test has Narrow input data, then it is supplying data for just part of the Input Data Model. Tests with narrow input data are less affected by Test Suite Degradation in the case of Input Data Model Evolution .Tests that focus on Narrow Regulatory Logic can make use of Narrow Input Data.

Wide Input Data

If a test has wide input data, then it is supplying data for most of the Input Data Model, though perhaps limited to one asset class. This allows for calculation of a wider range of end results (e.g. market risk, credit risk liquidity risk) and requires wider knowledge in setting the input fields (e.g. modified duration for the duration approach to market risk, probability of default for IRB credit risk, and cashflow for liquidity reporting) , Wide input data is more affected by Test Suite Degradation in the case of Input Data Model Evolution

Skinny Input Data

Skinny input data is where we try to focus on providing one or two trade/position/transaction as input data for a test. Failure of a test with skinny input data gives very specific information on what failed

Fat Input Data

Fat Input data is where in one test we provide a large set of trades/transaction/positions covering a wider range of inputs into one test. Failure might give less specific information. Fata Input Data can be useful in Regression Tests. As is is likely to cover more processing and more input data types Fat Input Data is more affected by Test Suite Degradation as in the case of Input Data Model Evolution and Regulatory Logic Evolution.

Narrow Expected Results

Expected Results which cover a small set of results of the Output Data Model. For example the expected result of one report cell. This can be good for Narrow Regulatory Logic, and Test Accountability when an expert knows one set of results well. The combination of Narrow Expected Results and Narrow Regulatory Logic can give clarity to Test Coverage as it can focus on covering just one scenario in a Scenario Taxonomy (e.g market risk capital charge, for bonds, using the modified duration approach). It is less affected by Test Suite Degradation as a result of Report Structure Evolution . If a small output is likely to be executing less of the Regulatory Logic it is also less affected by Test Suite Degradation as a result of Regulatory Logic Evolution .

Wide Expected Results

Wide Expected Results are expected results which populate a wide range of items in the Output Data Model . These can be useful for Regression Tests . Wide Expected Results are testing more paths through the regulatory logic and so are more affected by Test Suite Degradation as a result of Regulatory Logic Evolution and Report Structure Evolution. Wide expected Results can sometimes pollute a test by executing logic that is not in the primary purpose of a test.

Test Accountability

Testing is difficult and requires a mixture of regulatory knowledge and knowledge of data structures. A single test can evolve in a haphazard way if not controlled, and the reason for a particular test may go undocumented, and it may be tricky to say who is the owner or authority on a test. This can lead to a lack of trust in a test suite. For single tests which test Narrow Regulatory Logic and provide Narrow Expected Results with Skinny Input Data, and Expected Results aimed a testing one specific Test Scenario in a Test Scenario Taxonomy(e.g. Credit Risk for CDS with Securities netting using the comprehensive approach to credit Risk mitigation ) are easier to assign to someone who is an expert in that exact field (who is more likely to accept accountability as the test only tests what they are expert in). Test Review by other experts in that specific field also provides a shared accountability, and greater confidence in a test.

Test Review

Test Review provides a means for different eyes to give views on a test, this is easier when the test is on Narrow Regulatory Logic

Narrow Regulatory Logic

Regulatory Logic can be considered as many possible paths through a set of logic resulting in multiple outputs (e.g. report cells). Within regulatory logic there are often many conditional aspects (e.g. do this for an-balance, that for off-balance) and often optional choices (e.g. the simple or comprehensive approach to credit risk mitigation) . Many of these paths are executed when we run regulatory logic. A test for narrow regulatory logic aims a to isolate one path or one small set of possible paths. A test of Narrow Regulatory Logic with Narrow Output Test Data is good at ticking offone precise part of a detailed Test Scenario Taxonomy with confidence IF it has Expected Results as Domain Tests. Just knowing that an execution path run may be useful, but not knowing if the result is correct (or just knowing that it gave the same result as last time) does not provide faith in the test.

Test Coverage

Test Coverage is a measure against a Test Scenario Taxonomy which reflects how much of the taxonomy has associated tests.

Calculation Regulatory Logic

Some regulatory logic defines how calculations should be done (like the calculation of market price)

Workflow Regulatory Logic

Some regulatory logic defines how which flow of processing should be done, like which set of credit risk processing logic should be applied, dependent upon the exposure class (one set of logic for government exposures, one for corporate exposures etc.)

Test Suite

A test suite is a group of tests

Test Suite Degradation

Test Suites start to contain invalid test over time if they are not carefully maintained at the same time as Input Data Models, Regulatory Logic, Test Scenario Taxonomies, and Output Data Structures are evolving.

Test Validation

The validation of a test to check if it is referring to Input Data Model items, Output Data Structure items, or Test Scenarios which are no longer existing or have had important changes in meaning that could affect any associated Expected Results.

Partial Test Suite Degradation

As structures and logic and requirements change it is highly likely that some tests in a test suite become invalid but not all tests. It is good to know which are still valid in order to maintain faith in a test suite. The rate of degradation depends on the testing approach as some are more affected by evolution of structures and logic and requirements

Input Data Model Evolution

Input Data models change over time, this is more likely in a Harmonised Data Model than a Specific Data Model

Report Structure Evolution

Report Structures Change over time

Regulatory Text evolution

Regulatory text evolves over time with changes made by regulators

Regulatory Logic Evolution

As Regulatory Text changes, so Regulatory Logic changes

Scenario Taxonomy Evolution

As Regulatory Logic changes, so the Test Scenario Taxonomy should change to reflect the new logic

Cascading Dependencies

Regulatory text changes affect Regulatory Logic, which affects Test Scenario Taxonomies which affect Test Definitions, which (along with Input Data Model changes and Report Structure changes) affect tests. Importantly at each stage of this cascade we only invalidate some of the next change. The addition of one line of regulatory text is unlikely to invalidate all of a scenario taxonomy. It is good (and possible) to determine what is affected and what is not.

Test Data Re-use

Multiple tests might reuse the same data, or similar data

Example Test data

It is useful to have some reusable snippets of Input Test Data which are useful across tests, such as a standard loan

Wide Test Example Data

A Template may choose to have the input fields relevant to many different scenarios (like market risk, liquidity risk, and counterparty risk), but this requires expertise in a large part of the Input Data Model. Or it could have just some.

Immutable Test Data

It is important that if multiple Testsuse the same Input Test Data that the effects of changing that single piece of Test Data are understood. An easy way of manging this is to say that the shared test data is never changed. In this case a new version of the example data could be made but existing tests should still point to the unchanged previous version. It may be advisable to decide which tests should be carefully changed explicitly to point to the new version of some Input test Data.

Cookie Cutter Input Data

It is possible to have a set of Example Test Data (wide or narrow) which acts as useful example data but is never referred to directly. In this case we can use that example data as a template, copy it and amend the copy as we like and use the copied amended unchangeable version in our tests. Note that as and Input Data Model changes, the useful example data may become out of data if not maintained properly in line with the Input Data Model.

Versioned Test Suites

It makes sense for test suites to have versions. It can be quite acceptable that some of the tests in a new version of a test suite are test from the previous version, and not just copies of tests from a previous version (i.e. they retain the same ID)

Centrally Managed Test Data

It can be useful to maintain data centrally in a way that is accessible to testers, domain experts and developers. Tests that are not centrally available because they are combined into the technical code base can be difficult for no technical users to make use of and to provide feedback upon.

Test with Live Data

Exports of live data from banking system can be useful input data as they are likely to cover the range of products that a bank uses. This can be useful as a Regression test. However, it suffers from some of the same drawback of Fat Data if used as one block. If split into smaller pieces, it may be hard to assign to a specific point in a Test Scenario Taxonomy, as the test is not designed for that Test Scenario. If not further cut down it may suffer form some disadvantage of Wide Input Data .

If the live data came from actual regulatory systems, then the output may be useful for testing a different regulatory system , which is best if it can be associated with a scenario in a taxonomy. This may require mapping of different Input Data Models which may be time consuming. Live test data may make good input for Cookie Cutter Input Data

Tests Without Defined Data Model

We may have the case where we have a Test Scenario but we do not have a complete (or even any) data model, this makes the provision of Test Input Data difficult. In this case a speculative model may be of some use using well know terms like currency, interest rate, counterparty ID, but his has limitation, especially with any form of automated testing such as continuous integration.