Conformance - ChristianPresley/zig-sdk GitHub Wiki
The official conformance suite is the npm package @modelcontextprotocol/conformance1. The CI job conformance runs the pinned version 0.2.0-alpha.11 on each push to main and on each pull request. The fixtures are in conformance/ in the repository. The requirement matrix maps each normative sentence of the specification to a test.
Applies to: MCP 2026-07-28, zig-sdk 0.1.0. The scenarios of the authorization extensions, the unscored scenarios in CI, the authorization server scenarios, the typed gRPC binding and the requirement matrix: zig-sdk 0.2.0.
| Suite | Scenarios | Result |
|---|---|---|
Server, --requirements 2026-07-28
|
37 scored | All pass. The baseline is empty. |
Client, --requirements 2026-07-28
|
32 scored (7 protocol, 25 authorization) | All pass. The baseline is empty. |
| Not scored, server: Tasks extension | 10 | All functional checks pass. The check wire-schema-valid fails in the 8 scenarios that create a task, see below. |
| Not scored, server: pending | 3 | All pass: json-schema-2020-12 (8 checks), http-header-validation (14 checks) and http-custom-header-server-validation (10 checks). |
| Not scored, client: authorization extensions | 6 | All pass: auth/client-credentials-jwt, auth/client-credentials-basic, auth/enterprise-managed-authorization, auth/dpop, auth/dpop-nonce and auth/wif-jwt-bearer. |
Authorization server, the authorization command |
2 | Both pass: authorization-server-metadata-endpoint (2 checks) and authorization-code-grant (1 check). |
| Not scored, client: added after the release | 1 |
json-schema-2020-12-preservation passes all 9 checks. The client keeps $schema, $defs, $anchor, the composition keywords and the conditional keywords of a tool schema. |
The suite does not score a scenario of an extension, a pending scenario and a scenario that came after the release of the requirements. Its requirements file names each of them with the reason.
The tests of the SDK also run the draft 2020-12 files of the JSON Schema Test Suite against the validator. See JSON Schema Support.
The check wire-schema-valid validates each tools/call result against the CallToolResult schema of the core specification. That schema requires content. A CreateTaskResult of the Tasks extension has no content, so the check fails for each of the 8 scenarios that create a task. The scenario tasks-status-notifications has no checks, and tasks-required-task-error creates no task. See Tasks Extension.
The fixture client uses ClientCredentials for the two client credentials scenarios and EnterpriseClient for the enterprise-managed authorization scenario. It uses WorkloadIdentity with the workload JWT of the scenario for auth/wif-jwt-bearer. For auth/dpop and auth/dpop-nonce, it gives OAuthClient a new DPoP key. See Authorization.
The --requirements 2026-07-28 runs include the unscored scenarios, but their failures do not change the exit code. Thus the CI job also runs each of the 20 unscored scenarios alone. It uses --force and the baseline conformance/expected-failures-unscored.yml. An entry of that baseline names one check of one scenario: the 8 wire-schema-valid checks of the Tasks scenarios. Any other failure makes the job fail, and so does a listed check that passes.
To run one unscored scenario alone, use --scenario with --spec-version 2026-07-28 --force instead of --requirements. The suite does not accept --requirements together with these options. Without --force, the suite skips an extension scenario:
npx --yes @modelcontextprotocol/[email protected] client --command ./zig-out/bin/mcp-conformance-client --scenario auth/dpop-nonce --spec-version 2026-07-28 --force --expected-failures conformance/expected-failures-unscored.ymlThe authorization command of the suite tests an authorization server. The CI job conformance runs it against the fixture mcp-conformance-authorization-server, which is the authorization server of the SDK with one pre-registered public client. The fixture approves each request for one test user. The suite prints the authorization URL and waits for a browser, so the CI job opens the URL with curl -L. The redirect then goes to the callback of the suite.
To run the scenarios on your computer, build the fixture with zig build conformance-authorization-server. Start it with ./zig-out/bin/mcp-conformance-authorization-server --port 3100. Then run the suite, and open the authorization URL that it prints:
npx --yes @modelcontextprotocol/[email protected] authorization --url http://127.0.0.1:3100 --client-id conformance-client --port 3101Do not give --scenario. The scenario authorization-code-grant needs the result of authorization-server-metadata-endpoint, and only a run of all scenarios gives it. See Authorization Server.
The official suite has no gRPC scenarios. The CI job grpc-interop starts the fixture server with --grpc-port and --grpc-typed and calls it from three foreign clients. A Node script on the built-in http2 module (.github/interop/h2_peer.mjs) checks discovery, tools/list, the error trailers and a listen stream. @grpc/grpc-js calls all eight RPCs of the typed service. Then curl with HTTP/2 prior knowledge sends one raw call. See Interoperability.
- Build the fixtures:
zig build conformance-server conformance-client. - Start the server:
./zig-out/bin/mcp-conformance-server --port 3000. - Run the server suite:
npx --yes @modelcontextprotocol/[email protected] server --url http://127.0.0.1:3000/mcp --requirements 2026-07-28 --expected-failures conformance/expected-failures-server.yml- Run the client suite. On Windows, give the command as an absolute path:
npx --yes @modelcontextprotocol/[email protected] client --command ./zig-out/bin/mcp-conformance-client --requirements 2026-07-28 --timeout 60000 --expected-failures conformance/expected-failures-client.ymlThe requirement matrix maps each normative sentence of the specification to a test. A normative sentence has a key word of RFC 21192 in uppercase. The matrix is the generated file docs/generated/conformance-matrix.md in the repository. The sentences come from the source pages of the specification at a fixed commit3.
Each sentence has one status:
| Status | Meaning |
|---|---|
| tested | A Zig test or a scenario of the conformance suite checks the sentence. |
| n/a | The sentence does not apply to the SDK. Examples are rules for earlier protocol versions and rules for a proxy or an intermediary. |
| app | The application that uses the SDK must do it. Examples are user consent, the user interface and the policy of a tool. The reason names the hook or the option that the SDK gives. |
| gap | No test checks the sentence yet. The reason tells whether the behavior is partial or missing. |
In release 0.2.0, the matrix has 733 sentences: 488 tested, 50 n/a, 195 app and 0 gaps. 393 sentences have the key words "MUST" and "MUST NOT". Of these, 299 have a test, and 94 have a reason for the status n/a or app. The obligations of an authorization server map to the tests of AuthorizationServer and to the authorization server scenarios. No "MUST" or "MUST NOT" sentence is a gap. The summary of the matrix file docs/generated/conformance-matrix.md has the current numbers by keyword and by page.
-
zig build extract-requirementsreads the specification pages intest/fixtures/mcp_spec_2026_07_28/. It writes each normative sentence todocs/spec/requirements.zonwith a stable id, the key word and the party. - The file
docs/spec/requirement_tests.zonmaps each id to tests, or to a status with a reason. We keep this file by hand. -
zig build spec-matrixchecks this file and writes the matrix. It fails when a sentence has no entry, when an id is stale or when a test does not exist.
The CI job fmt-lint runs both steps with --check. Thus a change to the specification pages or to a test name needs a change to docs/spec/requirement_tests.zon.
The extraction uses the prose pages of the specification. It does not read the comments of schema.ts or the changelog page.
The suite defines conformance tiers. Tier 1 as the suite defines it today requires revision 2025-11-25 as well. This SDK obeys revision 2026-07-28 only, so it does not claim Tier 1. It publishes the score of the --requirements 2026-07-28 run instead.
conformance/expected-failures-server.yml and conformance/expected-failures-client.yml list known failures of the scored scenarios. Both lists are empty. conformance/expected-failures-unscored.yml lists the 8 wire-schema-valid checks of the Tasks scenarios. A listed failure that passes makes the CI job fail, so the lists cannot become stale.
- MCP-CONFORMANCE: The MCP conformance test suite.
- RFC2119: "Key words for use in RFCs to Indicate Requirement Levels".
- MCP-SPEC-SOURCE-2026-07-28: The source pages of the specification at the vendored commit.