Comparison - TestlumFramework/Testlum GitHub Wiki

Comparison in Testlum

Testlum has a function of comparison.

This function implies comparing the expected result with the actual one after the step is completed.

To compare test results, the following files are used:

expected_n.json - expected test result actual_n.json - actual test result

Having the structure expected_1.json, actual_1.json - the number is put, depending on the step of the test scenario.

Presence of expected file is a mandatory parameter for HTTP, SQL and noSQL requests.

The principle of operation on the example of postgres:

<postgres comment="Check successfully adding Product to shopping cart"
          alias="Shop" file="expected_11.json">
    <query>
        SELECT shp_cart_id, customer_id, shp_cart_code
        FROM shopping_cart
        WHERE merchant_id = 1
    </query>
</postgres>

Steps

  • Make a request specifying the expected file (with the scenario step number, in this case expected_11.json).
  • Create a file in the scenario folder with the name specified inside the postgres request (in this case, expected_11.json).
  • Leave generated expected_11.json empty.
  • Run test scenario.
  • After running the test scenario and executing this query, comparison will automatically generate an actual_file with the scenario step number, see an empty file expected_11.json and compare it with the query result received in actual_11.json. If the result of the request is satisfactory to the user, it will transfer all data from the actual file to the expected file, for further comparison and successful completion of the test scenario.
  • comparison - will generate the actual file only if the content between actual and expected does not match.

Comparison modes

Testlum supports two modes for comparison. Let's dive deeper into strict and lenient comparison modes in Testlum.

strict mode

Strict mode means everything must match exactly — including:

  • All fields must be present.
  • Order of elements (especially in arrays/lists) must match.
  • Data types must be exactly the same.
  • Values must be identical.
<http comment="Check ability to login added user with valid credentials" 
      alias="MEGA_APP_BASIC">
    <post endpoint="/basic/login">
        <response code="200" file="expected_3.json" mode="strict"/>
        <header name="Authorization"
                data="Basic dGVzdGx1b"/>
        <body>
            <raw>
                {
                "username": "testlum",
                "password": "$2a$10$S"
                }
            </raw>
        </body>
    </post>
</http>

lenient mode

Lenient mode is more forgiving:

  • Extra fields are usually ignored.
  • Order may not matter (especially in JSON objects or arrays).
  • Minor type mismatches may be tolerated (e.g., "123" vs. 123).
  • Only relevant or expected fields are compared.
<http comment="Check ability to login added user with valid credentials" 
      alias="MEGA_APP_BASIC">
    <post endpoint="/basic/login">
        <response code="200" file="expected_3.json" mode="lenient"/>
        <header name="Authorization"
                data="Basic dGVzdGx1b"/>
        <body>
            <raw>
                {
                "username": "testlum",
                "password": "$2a$10$S"
                }
            </raw>
        </body>
    </post>
</http>

✅ Tips:

strict mode is useful for verifying exact structure (e.g., schema validation).

lenient mode is better for checking essential correctness without enforcing formatting or extras.

strict is default mode for comparison in Testlum

Comparison Modes Summary

Feature Strict Mode Lenient Mode
Field presence All fields must be present Missing fields are tolerated
Field order Must match exactly (especially for arrays) Order can differ
Data type Must match exactly (e.g., "123" vs 123 considered different) Minor mismatches allowed (e.g., "123" vs 123 considered equal)
Extra fields Not allowed Ignored
Default mode ✅ Yes ❌ No
Suitable for Schema validation, exact structural match Functional correctness without strict enforcement
Example usage <response code="200" file="expected.json" mode="strict"/> <response code="200" file="expected.json" mode="lenient"/>
Regular Expressions

While comparing actual and expected files, some dynamical values might appear. For instance, in actual file with results of HTTP or SQL query (IDs, timestamp, etc). To avoid exception while comparison, you can use the following list of regular expression:

Name Pattern
p(any) any sequence of any symbols
p(digit) integer value (1,2,3,4,5...)
p(money) float value (100.10, 532.83...)
p(email) [email protected]
p(ip) 10.10.10.10
p(url) http://google.com
p(uuid) 550e8400-e29b-41d4-a716-446655440000
p(color) #1f1f1f
p(y-m-d) 2015-11-12
p(d-m-y) 12-11-2015
p(notEmpty) any symbols but not empty

Here is an example of using regular expressions in expected file:

{
  "id": "p(digit)",
  "token": "p(any)"
}

Also, we can set condition for our expected value. For instance:

Condition Description
c(>100) any integer value more than 100
c(>-111.11) any float value more than -111.11
c(<=145387) any integer value lower than or equal to 145387
c(>2001-02-16 20:00:00) any timestamp after 2001-02-16 20:00:00
c(<=now) any timestamp before or equal to the current timestamp
Symbols escaping

To use reserved symbols in scenario or json files, you need to escape them.

Here is the list of symbols you need to escape:

  • "
  • .
  • *
  • ^
  • $
  • ?
  • +
  • -
  • {}
  • []

The way of symbol escaping depends on where you need to escape them. If you need to escape symbol in JSON file or in scenario, you need to follow the rules of escaping for specific format you're using (JSON, XML and so on).

So, here are some examples of symbol escaping for JSON and XML formats:

{
"query": "{ getBookByTitle(title: \"A Man Called Ove\") { id number } }"
}
<graphql comment="GraphQL query with invalid operationName for Get method" alias="spaceX">
        <get endpoint="/graphql">
            <response code="400" file="expected_6.json"/>
            <param name="query"
                   data="query GetLaunchesByMissionName($missionName: String!) { launches(find:{
                         mission_name: $missionName }) { mission_name launch_date_local launch_success                            rocket {rocket_name rocket_type}}}"/>
            <param name="variables" data="{ &quot;missionName&quot;: &quot;CRS-16&quot; }"/>
            <param name="operationName" data="GetLaunches"/>
        </get>
</graphql>
⚠️ **GitHub.com Fallback** ⚠️