Data Template Payload Validation

1. Overview

To improve data integrity and completeness, we have implemented a robust validation system for payloads submitted to our data templates endpoint. This system checks incoming data against a predefined set of rules before it is processed.

The primary goals of this feature are:

• Enforce Data Contracts: Ensure that all required fields are present and correctly formatted.
• Improve Data Quality: Prevent incomplete or malformed data from entering our system.
• Provide Clear Feedback: Return specific, actionable error messages to the client when validation fails.

This validation logic is powered by the Spring Expression Language (SpEL), a powerful expression language that allows for querying and manipulating an object graph at runtime.

2. How It Works

When a payload is sent to a data template endpoint, the following process occurs:

1. The system retrieves all active validation rules associated with that data template from the database.
2. For each rule, the SpEL rule_expression is evaluated against the incoming payload.
3. If an expression evaluates to false, the validation fails.
4. The system immediately stops processing and returns a 400 Bad Request response, including the corresponding code and error_message for the failed rule.
5. If all rules pass (evaluate to true), the payload is considered valid, and processing continues.

---

3. The Validation Rule Generator UI

To simplify the creation and management of these rules, we have developed a Validation Rule Generator interface. This tool provides a user-friendly way to define validation logic without needing to write raw database queries.

Key features of the UI:

• Guided Rule Creation: A step-by-step form for filling in all necessary fields for a validation rule.
• Rule Management: View, edit, activate, or deactivate existing rules.
• Syntax Highlighting: The expression editor provides helpful highlighting for SpEL syntax.
• (Optional) Live Testing: A "Test" feature allows you to paste a sample JSON payload and see if it passes or fails your rule expression in real-time.

Validation Rule Generator

---

4. Validation Rule Structure

Each validation rule is stored in our database and is defined by the following fields:

Column Name	Type	Required	Description
name	VARCHAR	Yes	A short, human-readable name for the validation rule (e.g., "Validate User Email"). Used for identification in the UI.
description	TEXT	No	A more detailed explanation of what the rule does and why it's necessary.
error_message	VARCHAR	Yes	The specific error message returned to the client if this rule fails. Should be clear and actionable (e.g., "User email is missing or not a valid format.").
code	VARCHAR	Yes	A unique error code associated with this validation failure (e.g., E_1001_INVALID_EMAIL). This allows client applications to programmatically handle specific errors.
rule_expression	TEXT	Yes	The core of the rule. A SpEL expression that must evaluate to true for the validation to pass. See the next section for details and examples.

---

5. Writing Rule Expressions with SpEL

The rule_expression uses SpEL to evaluate the incoming JSON payload. In your expressions, the entire payload is available as the root object, which you can reference using the # symbol (e.g., #{payload}).

Context:

• Root Object: The entire payload is accessible as #payload.
• Accessing Fields: Use dot notation to access nested fields (e.g., #{payload.customer.address.city}).
• Return Value: The expression must return a boolean (true or false). true means the validation passes.

Common Examples

Let's assume an incoming payload structure like this:

{
  "orderId": "ORD-12345",
  "customer": {
    "id": "CUST-007",
    "tier": "GOLD",
    "contact": {
      "email": "[email protected]"
    }
  },
  "lineItems": [
    { "sku": "SKU-A", "quantity": 2 },
    { "sku": "SKU-B", "quantity": 1 }
  ],
  "isPriority": true
}

---

Example 1: Check for Field Presence

Ensure the customer ID is not null.

• rule_expression: #{payload.customer.id} != null

---

Example 2: Check String Format (Regex)

Validate that the orderId starts with "ORD-".

• rule_expression: #{payload.orderId}.matches('^ORD-.*')

---

Example 3: Check a Numeric Value

Ensure a line item's quantity is greater than 0. This example uses collection projection to check all items in a list.

• rule_expression: #{payload.lineItems}.?[quantity <= 0].isEmpty()
  • Explanation: This expression filters the lineItems list to find any items where quantity is less than or equal to 0. It then checks if the resulting list is empty. If it's empty, it means no invalid items were found, so the rule passes (true).

---

Example 4: Check for List/Collection Emptiness

Ensure the lineItems array is not empty.

• rule_expression: !#payload.lineItems.isEmpty()

---

Example 5: Conditional Logic (AND / OR)

A high-priority order for a GOLD tier customer must have an email address.

• rule_expression: !#{payload.isPriority} or #{payload.customer.tier} != 'GOLD' or #{payload.customer.contact.email} != null
  • Explanation (using De Morgan's laws for clarity): This rule fails only if isPriority is true AND customer.tier is GOLD AND email is null. In all other cases, it passes.

---

A Complete Rule Example

Here is how a complete rule to validate the customer's email would look in the database.

• name: Validate Customer Email
• description: Ensures that every payload contains a customer email and that it follows a basic email format.
• error_message: Customer email is missing or is not a valid email format.
• code: E_1002_INVALID_EMAIL
• rule_expression: #{payload.customer.contact.email} != null and #{payload.customer.contact.email}.matches('^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,6}$')c