RDK Engineering Validation Framework - rdkcentral/rdk-halif-aidl GitHub Wiki

RDK’s validation ecosystem is founded on deterministic engineering principles. Every component, group, and layer must be tested, approved, and documented before integration.

Two complementary validation domains exist:

• Engineering-Led Deterministic Testing (VTS/MTS/ATS): Pre-commit validation ensuring that all code changes meet engineering requirements and design intent.

• Post-Commit Automation (Automatics): E2E system-level regression and experience validation executed via CI/CD after code integration.

This architecture enforces a bottom-up validation flow — determinism first, automation last.

flowchart TB
  subgraph GOV["Governance"]
    AB["Architecture Board"]
    QA["QA / Release Mgmt"]
  end

  subgraph REQ["Architecture & Requirements"]
    AR["Requirements & Design Reviews"]
    TR["Test Requirements<br>(Deterministic Criteria)"]
  end

  AB --> AR --> TR

  subgraph ENG["Engineering Teams"]
    direction TB
    subgraph VENDOR["Vendor Layer"]
      VTS_C["Component Tests<br>(HAL / Drivers APIs)"]
      VTS_G["Group Tests<br>(Composite HAL: Tuner + Demux + Decoder)"]
    end
    subgraph MIDDLE["Middleware Layer"]
      MTS_C["Component Tests<br>(Services / IPC / Framework Modules)"]
      MTS_G["Group Tests<br>(Service Interactions & IPC Contracts)"]
    end
    subgraph APP["Application Layer"]
      ATS_C["Component Tests<br>(App Logic, State Machines)"]
      ATS_G["Test-Driven Apps<br>(Full-stack Stress & Workflows)"]
    end
  end

  subgraph PRE["Pre-Commit Validation Gate"]
    PC["Deterministic Execution<br>Known Versions - Known Configs<br>Traceable Results - Coverage Met"]
  end

  subgraph CI["Post-Commit System Automation"]
    AUTO["Automations<br>E2E Integration - Regression - UX Signals"]
  end

  TR --> ENG
  VTS_C --> VTS_G --> PC
  MTS_C --> MTS_G --> PC
  ATS_C --> ATS_G --> PC
  PC -->|Merge| AUTO --> QA

  AUTO -. "integration feedback" .-> ENG
  ENG -. "test coverage & gaps" .-> TR
  TR -. "periodic review" .-> AB
  QA -. "release feedback" .-> AB

Engineers must:

• Execute VTS/MTS/ATS suites relevant to their components.
• Upgrade suites when introducing new functionality.
• Maintain traceability between requirements, code, and test coverage.
• Document all deterministic testing activity in versioned form.

Testing is a core engineering responsibility, not a CI function.

sequenceDiagram
  autonumber
  participant Dev as Engineer
  participant VTS as VTS/MTS/ATS Suites
  participant Docs as Test Docs & Traceability
  participant PR as Pull Request
  participant Rev as Reviewers/Leads
  participant CI as Automatics (Post-Commit)
  participant QA as Release Mgmt

  Dev->>VTS: Run deterministic component & group tests
  Dev->>Docs: Update specs, coverage, trace matrix
  Dev->>PR: Submit PR with logs/reports & updated docs
  Rev->>PR: Review test evidence & documentation
  Rev->>VTS: Request extensions if coverage incomplete
  VTS-->>Dev: Identify gaps or missing tests
  Dev->>PR: Update test suites & resubmit
  Rev->>PR: Approve (criteria satisfied)
  PR->>CI: Merge triggers Automatics regression & E2E validation
  CI-->>Dev: Integration feedback & regressions
  CI-->>QA: Build health reports for release gate

• Reviewers ensure test suites are executed and documented.
• New functionality must extend test suites where applicable.
• PR merges only when deterministic validation is complete.

Test documentation evolves with code:

• Test specifications and expected results must be kept current.
• Each change to code requires corresponding test updates.
• A traceability matrix links requirements ↔ code ↔ tests ↔ results.
• Historical version control identifies applicable suite versions.

Test suite documentation is an engineering artifact. It provides traceable proof of deterministic validation.

Changes to controlled interfaces (e.g., HAL, AIDL, Thunder APIs):

• Follow formal architecture review and approval.
• Require versioning and backward compatibility validation.
• Trigger related updates in VTS/MTS/ATS test suites.

While detailed API governance lies outside this document, all interface changes must adhere to engineering governance policies for consistency and determinism.

Each suite validates determinism within its scope, building trust layer by layer without redundancy.

• VTS/MTS/ATS must never be executed through CI/CD.
• Operate in controlled, version-locked environments.
• Each test must be reproducible and traceable.
• Known configurations, dependencies, and expected results are mandatory.
• Execution evidence feeds back into documentation and architecture reviews.

CI validates integration; engineering testing validates design correctness.

• Executes E2E regression and integration validation.
• Manages Jenkins orchestration, device management, and CI gating.
• Verifies build health and customer experience quality.

Automatics assumes that all components have already been engineering-validated. It confirms that deterministically proven modules integrate cohesively into a stable system.

flowchart LR
  RQ["Requirement ID(s)"] --> DS["Design Spec / Interface Contract"]
  DS --> TS["Test Specification<br>(coverage, parameters, expected results)"]
  TS --> TC["Test Code<br>(VTS / MTS / ATS Implementation)"]
  TC --> TR["Execution Results<br>(logs, reports, traces)"]
  TR --> TM["Trace Matrix<br>Req <-> Code <-> Tests <-> Results"]
  TM --> VR["Versioned Release<br>(Tag / Commit / Build)"]

  AB["Architecture Board"] --> DS
  AB --> TS
  QA["QA / Release"] --> VR

  TR -. "feedback" .-> TS
  TS -. "suite upgrades" .-> TC

Traceability ensures that requirements, design, code, and validation remain synchronized throughout the product lifecycle.

Feedback Loops:

1. Architecture → Engineering: defines design/test criteria.
2. Engineering → Validation: executes suites, updates documentation.
3. Validation → Architecture: feeds results for design evolution.
4. Automation → Engineering: reports integration regressions for engineering correction.