Test Case Generation ‐ Positive and Negative and Combined Tests - Gnorion/BizVR GitHub Wiki

POSITIVE/NEGATIVE TEST CASES

Looks like we can do this.

Basically all the + test cases are generated from the values actually tested in the rules

And all the -ve test cases can be generated from all the gaps between the values tested in the rules (i.e. the values that are not tested)

Ideally if the rules are complete there will be no -ve test cases.

Example

For this table

image

Here is the + test case (a sample value that is in the range)

image

The - test cases would be one value below the range and one value above the range

image

But there's also another category of test cases - the exceptions - where the values are unexpected (such as " " for a numeric field)

Still need to figure out what value (if any) make sense to test here

When the rules get a bit more complex it starts to look like this

image

positive test cases (one value for age x 2 values for income)

image

negative test cases (2 missing values for age x one missing value for income 1000 exactly)

image

COMBINED POSITIVE AND NEGATIVE TESTS

If we merged the + and - test cases into a single set then we'd get 9 test cases. (3 values of age x 3 values of income).

image image image

Note only tests 1,2,5,8 will produce output since there are actual rules for those cases. The others are not covered by the rules.

So generating separate sets will reduce the number of test cases required but its no longer testing every possible combination.

User must decide what is important.

Also if the user decides additional values would be useful they can easily add them by creating a "seeding" table in the model that just tests the attribute against any value they think is important. Those values will automatically be included in the text case generation.

For example a table like this will add values 10 and 16 for variable "a" and values 5 and 15 for variable "b" in addition to whatever the business rules are testing.

image

This technique can also be used when the rules don't actually have any literal values (such as the eHarmony or scheduling models) and are simply comparing values supplied in the input data. VV cannot generate test cases from the rules since there are no values. But it can use the seeding data to generate cases since that contains actual literal values.

MORE THOUGHTS ON TEST CASES

There may be several options for the user to pick from when generating test cases:

  1. no text cases
  2. positive case only - generated from the actual values used in the rules
  3. negative cases only - generated from the gaps in the values used in the rules
  4. both cases combined - all combinations of every possible value (may be very large)
  5. exception cases (user needs to specify the exception values to be considered)
  6. Seeded values (user optionally provides extra values for testing
  7. Pinned values (user narrows the choice of values for some attributes e.g only test cases where age=18)
  8. Test cases generated by rule (i.e. one test case per rule). This would be minimal + cases
  9. Test cases generated by missing rule (i.e test case for each missing rule). Minimal - cases

we may need to experiment with this a bit to see what makes most sense

The missing rules analysis is based on any gaps VV finds in the values being tested in the actual rules.

The difference is that for rule based tcg we don't try to generate every possible combination - just one set of values that will fire the rule.

Currently we generate all combinations (which can result in a lot of test cases)

TCG by rule will has less test cases but will be less rigorous. May be a good place for the user to start testing.

Another Example of the Difference Between Positive, Negative and Combined Tests

image

image

image

Number of positive test required  = 2 [age(1) x income(2) x dob(1)]
Number of negative tests required = 4 [age(2) x income(1) x dob(2)]
Number of combined tests required =27 [age(3) x income(3) x dob(3)]