Conformance - ChristianPresley/zig-sdk GitHub Wiki

Conformance

The official conformance suite is the npm package @modelcontextprotocol/conformance1. The CI job conformance runs the pinned version 0.2.0-alpha.11 on each push to main and on each pull request. The fixtures are in conformance/ in the repository. The requirement matrix maps each normative sentence of the specification to a test.

Applies to: MCP 2026-07-28, zig-sdk 0.1.0. The scenarios of the authorization extensions, the unscored scenarios in CI, the authorization server scenarios, the typed gRPC binding and the requirement matrix: zig-sdk 0.2.0.

Results

Suite Scenarios Result
Server, --requirements 2026-07-28 37 scored All pass. The baseline is empty.
Client, --requirements 2026-07-28 32 scored (7 protocol, 25 authorization) All pass. The baseline is empty.
Not scored, server: Tasks extension 10 All functional checks pass. The check wire-schema-valid fails in the 8 scenarios that create a task, see below.
Not scored, server: pending 3 All pass: json-schema-2020-12 (8 checks), http-header-validation (14 checks) and http-custom-header-server-validation (10 checks).
Not scored, client: authorization extensions 6 All pass: auth/client-credentials-jwt, auth/client-credentials-basic, auth/enterprise-managed-authorization, auth/dpop, auth/dpop-nonce and auth/wif-jwt-bearer.
Authorization server, the authorization command 2 Both pass: authorization-server-metadata-endpoint (2 checks) and authorization-code-grant (1 check).
Not scored, client: added after the release 1 json-schema-2020-12-preservation passes all 9 checks. The client keeps $schema, $defs, $anchor, the composition keywords and the conditional keywords of a tool schema.

The suite does not score a scenario of an extension, a pending scenario and a scenario that came after the release of the requirements. Its requirements file names each of them with the reason.

The tests of the SDK also run the draft 2020-12 files of the JSON Schema Test Suite against the validator. See JSON Schema Support.

The check wire-schema-valid validates each tools/call result against the CallToolResult schema of the core specification. That schema requires content. A CreateTaskResult of the Tasks extension has no content, so the check fails for each of the 8 scenarios that create a task. The scenario tasks-status-notifications has no checks, and tasks-required-task-error creates no task. See Tasks Extension.

The fixture client uses ClientCredentials for the two client credentials scenarios and EnterpriseClient for the enterprise-managed authorization scenario. It uses WorkloadIdentity with the workload JWT of the scenario for auth/wif-jwt-bearer. For auth/dpop and auth/dpop-nonce, it gives OAuthClient a new DPoP key. See Authorization.

Unscored scenarios in CI

The --requirements 2026-07-28 runs include the unscored scenarios, but their failures do not change the exit code. Thus the CI job also runs each of the 20 unscored scenarios alone. It uses --force and the baseline conformance/expected-failures-unscored.yml. An entry of that baseline names one check of one scenario: the 8 wire-schema-valid checks of the Tasks scenarios. Any other failure makes the job fail, and so does a listed check that passes.

To run one unscored scenario alone, use --scenario with --spec-version 2026-07-28 --force instead of --requirements. The suite does not accept --requirements together with these options. Without --force, the suite skips an extension scenario:

npx --yes @modelcontextprotocol/[email protected] client --command ./zig-out/bin/mcp-conformance-client --scenario auth/dpop-nonce --spec-version 2026-07-28 --force --expected-failures conformance/expected-failures-unscored.yml

Authorization server scenarios

The authorization command of the suite tests an authorization server. The CI job conformance runs it against the fixture mcp-conformance-authorization-server, which is the authorization server of the SDK with one pre-registered public client. The fixture approves each request for one test user. The suite prints the authorization URL and waits for a browser, so the CI job opens the URL with curl -L. The redirect then goes to the callback of the suite.

To run the scenarios on your computer, build the fixture with zig build conformance-authorization-server. Start it with ./zig-out/bin/mcp-conformance-authorization-server --port 3100. Then run the suite, and open the authorization URL that it prints:

npx --yes @modelcontextprotocol/[email protected] authorization --url http://127.0.0.1:3100 --client-id conformance-client --port 3101

Do not give --scenario. The scenario authorization-code-grant needs the result of authorization-server-metadata-endpoint, and only a run of all scenarios gives it. See Authorization Server.

Interoperability of the gRPC binding

The official suite has no gRPC scenarios. The CI job grpc-interop starts the fixture server with --grpc-port and --grpc-typed and calls it from three foreign clients. A Node script on the built-in http2 module (.github/interop/h2_peer.mjs) checks discovery, tools/list, the error trailers and a listen stream. @grpc/grpc-js calls all eight RPCs of the typed service. Then curl with HTTP/2 prior knowledge sends one raw call. See Interoperability.

Steps: run the suite

  1. Build the fixtures: zig build conformance-server conformance-client.
  2. Start the server: ./zig-out/bin/mcp-conformance-server --port 3000.
  3. Run the server suite:
npx --yes @modelcontextprotocol/[email protected] server --url http://127.0.0.1:3000/mcp --requirements 2026-07-28 --expected-failures conformance/expected-failures-server.yml
  1. Run the client suite. On Windows, give the command as an absolute path:
npx --yes @modelcontextprotocol/[email protected] client --command ./zig-out/bin/mcp-conformance-client --requirements 2026-07-28 --timeout 60000 --expected-failures conformance/expected-failures-client.yml

Requirement matrix

The requirement matrix maps each normative sentence of the specification to a test. A normative sentence has a key word of RFC 21192 in uppercase. The matrix is the generated file docs/generated/conformance-matrix.md in the repository. The sentences come from the source pages of the specification at a fixed commit3.

Each sentence has one status:

Status Meaning
tested A Zig test or a scenario of the conformance suite checks the sentence.
n/a The sentence does not apply to the SDK. Examples are rules for earlier protocol versions and rules for a proxy or an intermediary.
app The application that uses the SDK must do it. Examples are user consent, the user interface and the policy of a tool. The reason names the hook or the option that the SDK gives.
gap No test checks the sentence yet. The reason tells whether the behavior is partial or missing.

In release 0.2.0, the matrix has 733 sentences: 488 tested, 50 n/a, 195 app and 0 gaps. 393 sentences have the key words "MUST" and "MUST NOT". Of these, 299 have a test, and 94 have a reason for the status n/a or app. The obligations of an authorization server map to the tests of AuthorizationServer and to the authorization server scenarios. No "MUST" or "MUST NOT" sentence is a gap. The summary of the matrix file docs/generated/conformance-matrix.md has the current numbers by keyword and by page.

How the matrix is made

  1. zig build extract-requirements reads the specification pages in test/fixtures/mcp_spec_2026_07_28/. It writes each normative sentence to docs/spec/requirements.zon with a stable id, the key word and the party.
  2. The file docs/spec/requirement_tests.zon maps each id to tests, or to a status with a reason. We keep this file by hand.
  3. zig build spec-matrix checks this file and writes the matrix. It fails when a sentence has no entry, when an id is stale or when a test does not exist.

The CI job fmt-lint runs both steps with --check. Thus a change to the specification pages or to a test name needs a change to docs/spec/requirement_tests.zon.

The extraction uses the prose pages of the specification. It does not read the comments of schema.ts or the changelog page.

Tier statement

The suite defines conformance tiers. Tier 1 as the suite defines it today requires revision 2025-11-25 as well. This SDK obeys revision 2026-07-28 only, so it does not claim Tier 1. It publishes the score of the --requirements 2026-07-28 run instead.

Baselines

conformance/expected-failures-server.yml and conformance/expected-failures-client.yml list known failures of the scored scenarios. Both lists are empty. conformance/expected-failures-unscored.yml lists the 8 wire-schema-valid checks of the Tasks scenarios. A listed failure that passes makes the CI job fail, so the lists cannot become stale.

References

  1. MCP-CONFORMANCE: The MCP conformance test suite.
  2. RFC2119: "Key words for use in RFCs to Indicate Requirement Levels".
  3. MCP-SPEC-SOURCE-2026-07-28: The source pages of the specification at the vendored commit.
⚠️ **GitHub.com Fallback** ⚠️