Platform Parity Check: AWS vs COTS — Post-Gap-Closure Assessment (2026-07-22)
Date: July 22, 2026
Operator: Terry McGuinness / Kiro AI
Purpose: Full-scope parity check with Phases 70–73 gap closures live on
COTS (container restarted, new code active). AWS runtime still on
python-tenants-v11 (pre-Phase-70). Confirms which gaps closed, surfaces new
deeper findings, and documents remaining actions.
Executive Summary
The COTS gap closures (Phases 70–73) are proven working. The Docker MCP
Gateway went from "3 integrity SKIPs + false Unhealthy + unlabeled counts" to a
trustworthy, honest, multi-language coverage report with real document counts
and self-describing scope annotations. The most impactful single fix (Phase 70:
ChromaDB count_documents + sample_metadata) unmasked a real Stale
Embeddings finding (50/50 docs >30 days) that was previously hidden by the SKIP —
the system is now honest rather than silently broken. AWS remains healthy but
still runs the old image (Phases 72–73 not yet visible there).
| Dimension |
AWS (AgentCore) |
COTS (Docker MCP Gateway) |
Parity |
| Overall health |
HEALTHY 4/4 |
HEALTHY 4/4 |
MATCH |
| Functional probes |
9/10 pass (1 SKIP: workflow_info/EFS) |
11/11 pass |
COTS leads |
| Vector docs (gw) |
252,013 |
220,538 |
AWS leads (+31K; dual-profile) |
| Graph nodes (gw) |
148,976 |
225,836 |
COTS leads (+77K; deeper Fortran) |
| Graph rels (gw) |
4,555,408 |
4,051,374 |
AWS leads (+504K; AST containment) |
| Whole-graph count |
(not available — old image) |
344,604 (Phase 73) |
COTS only |
| Integrity: Path |
[WARN] 2/34 |
[OK] 0/50 |
COTS leads (Phase 70) |
| Integrity: Stale |
[WARN] 12/12 |
[WARN] 50/50 |
Both stale (real finding) |
| Integrity: Coverage |
[SKIP] stale path |
[OK] 107K Fortran + Python + Shell |
COTS leads (Phase 72) |
| Integrity: Orphans |
[OK] 0/20 |
[OK] 0/20 |
MATCH |
setuprad traversal |
55 neighbors (degree 193) |
55 neighbors (degree 174) |
MATCH |
1. Gap Closures Confirmed Live on COTS
| Phase |
Gap (before) |
Result (after) |
Impact |
| 70 |
Total Documents: 0 [ERROR] Unhealthy |
Total Documents: 220,538 [OK] Healthy |
Real doc counts; false unhealthy eliminated |
| 70 |
[SKIP] adapter does not expose a metadata sampler (×2) |
Path: [OK] 0/50; Stale: [WARN] 50/50 |
Integrity checks execute; real findings unmasked |
| 72 |
[SKIP] no Fortran files found in ... |
[OK] 107,794 Fortran nodes for 7,242 files |
Multi-language coverage: Fortran + Python (8,607/41) + Shell (589/114) |
| 73 |
108280 nodes (unlabeled) |
108,280 nodes (health-check scope) |
Self-describing scope annotation |
| 73 |
No whole-graph visibility |
Total Nodes (all tenants, all labels): 344,604 |
Full graph via all_tenants=True |
| 73 |
Total Nodes: 225836 (unlabeled) |
Total Nodes (tenant gw): 225,836 |
Tenant-scoped, self-describing |
| 70 |
v17 Total Documents: 0 [ERROR] |
Total Documents: 2,610 [OK] Healthy |
PoC re-ingest data (92 jjobs + 2,518 docs) now visible |
2. Health Check Comparison
Component Status
| Component |
AWS |
COTS |
| Base Server |
[OK] FastMCP |
[OK] FastMCP |
| Utility Tools |
[OK] 4 tools |
[OK] 4 tools |
| Vector Database |
[OK] 21 indices |
[OK] 17 indices |
| Graph Database |
[OK] 105,891 nodes / 4,729,093 rels |
[OK] 108,280 nodes (health-check scope) / 4,229,217 rels |
Functional Validation
| Module |
AWS |
COTS |
| semantic_search |
[OK] 175ms |
[OK] 102ms |
| code_analysis |
[OK] 49ms |
[OK] 3ms |
| graph_rag |
[OK] 46ms |
[OK] 2ms |
| ee2_compliance |
[OK] 161ms |
[OK] 93ms |
| operational |
[OK] 188ms |
[OK] 89ms |
| sdd_workflow |
[OK] 0ms |
[OK] 0ms |
| workflow_info |
[SKIP] (EFS not mounted) |
[OK] 0ms |
| github_tools |
[OK] 200ms |
[OK] 104ms |
| utility |
[OK] 0ms |
[OK] 0ms |
| code_awareness |
N/A |
[OK] 28ms |
| branch_isolation |
[OK] 472ms |
[OK] 67ms |
3. Knowledge Base Status — Default Tenant (gw)
Vector Database
| Metric |
AWS (OpenSearch) |
COTS (ChromaDB) |
| Collections |
16 |
15 |
| Total Documents |
252,013 |
220,538 |
| Status |
[OK] Healthy |
[OK] Healthy |
| Serving profile |
titan1024 (1024-dim) |
mpnet768 (768-dim) |
COTS collection breakdown (Phase 70 now reporting real counts):
| Collection |
Documents |
Notes |
| mdc-workflow-docs-mpnet768 |
22,498 |
Serving docs |
| mdc-code-context-mpnet768 |
60,576 |
Serving code |
| mdc-jjobs-mpnet768 |
700 |
Serving jjobs |
| mdc-ee2-standards-mpnet768 |
34 |
Serving EE2 |
| mdc-community-summaries-mpnet768 |
2,113 |
Serving community |
| global-workflow-docs-v8-2-0 |
23,624 |
Legacy (v8.2) |
| global-workflow-docs-v8-1-0 |
20,511 |
Legacy (v8.1) |
| global-workflow-docs-v8-0-0 |
22,498 |
Legacy (v8.0) |
| code-with-context-v8-0-0 |
60,574 |
Legacy code |
| jjobs-v8-0-0 |
700 |
Legacy jjobs |
| jjobs-v8-1-0 |
859 |
Legacy jjobs v8.1 |
| ee2-standards-v5-0-0-enhanced |
34 |
Legacy EE2 |
| community-summaries |
2,113 |
Legacy community |
| ci-test-cases-v1-0-0 |
74 |
CI test cases |
| phase48-scratch |
3,630 |
Scratch (deletable) |
Graph Database
| Metric |
AWS (Neptune) |
COTS (Neo4j) |
| Total Nodes |
148,976 |
225,836 (tenant gw) / 344,604 (all tenants) |
| Total Rels |
4,555,408 |
4,051,374 |
| File |
17,273 |
17,273 |
| FortranSubroutine |
27,941 |
80,745 |
| FortranFunction |
5,744 |
16,849 |
| ShellScript |
315 |
589 |
4. Integrity Check Comparison
| Check |
AWS |
COTS |
Analysis |
| Path Consistency |
[WARN] 2/34 |
[OK] 0/50 |
COTS clean; AWS has 2 old-prefix docs (need re-ingest) |
| Orphaned Graph Nodes |
[OK] 0/20 |
[OK] 0/20 |
Both clean |
| Stale Embeddings |
[WARN] 12/12 |
[WARN] 50/50 |
Both stale — AWS 30+ days; COTS ~50+ days. Real finding on both; need re-ingest |
| Coverage Gap |
[SKIP] stale path |
[OK] 107,794 Fortran / 8,607 Python / 589 Shell |
Phase 72 transformed this from useless to actionable |
5. Knowledge Base Status — Tenant gw_v17
| Metric |
AWS (Neptune) |
COTS (Neo4j) |
Delta |
| File |
30,221 |
30,221 |
MATCH |
| FortranSubroutine |
36,156 |
36,156 |
MATCH |
| FortranFunction |
8,172 |
8,172 |
MATCH |
| FortranModule |
4,558 |
4,558 |
MATCH |
| ShellScript |
1,401 |
1,419 |
COTS +18 |
| Total Nodes |
80,996 |
81,014 |
COTS +18 |
| Total Rels |
1,278,331 |
1,279,922 |
COTS +1,591 |
| Vector docs |
56,876 |
2,610 |
AWS leads (full ingest vs partial PoC) |
6. Graph Traversal Parity: setuprad
| Metric |
AWS (Neptune) |
COTS (Neo4j) |
| Measured degree |
193 |
174 |
| Neighbors returned |
55 |
55 |
| Function overlap |
~98% identical |
~98% identical |
Both platforms return the same setuprad call-graph: qc_irsnd, qc_amsua,
qc_mhs, qc_atms, init_crtm, call_crtm, destroy_crtm, calc_clw,
ret_amsua, nc_diag_init, nc_diag_write, setup_sst_retrieval, get_radiag,
comp_fact10, dtime_setup, die, stop2, new, etc.
7. Health Trend (COTS — 10 snapshots)
| Date |
Neo4j Nodes |
Neo4j Rels |
ChromaDB Docs |
Collections |
Doc Drift |
| 2026-07-02 |
108,280 |
4,220,211 |
220,538 |
15 |
0 |
| 2026-07-09 22:07 |
108,280 |
4,229,217 |
223,148 |
17 |
+2,610 |
| 2026-07-10 – 2026-07-20 |
108,280 |
4,229,217 |
223,148 |
17 |
0 |
Trend: Graph nodes rock-stable (108,280 across 10 snapshots). Rels +9,006
(Jul 9 v17 shell-graph run). Docs +2,610 (Jul 9 v17 PoC). No anomalies.
8. Deeper Findings (New Gaps Surfaced)
| # |
Finding |
Platform |
Severity |
Resolution |
| D1 |
Stale Embeddings (50/50 on COTS; 12/12 on AWS) — now unmasked by Phase 70. All sampled docs >30 days old on both platforms. |
Both |
Medium |
Full re-ingest on both (framework spec for COTS; aws-doc-refresh-ingest for AWS) |
| D2 |
COTS v17 vector partial (2,610 vs AWS 56,876) — PoC only partially completed |
COTS |
Medium |
Framework spec's 44 pending units |
| D3 |
10 legacy collections on COTS still serving alongside mdc-* collections |
COTS |
Low |
Post-framework cutover + retirement |
| D4 |
AWS runtime on python-tenants-v11 not v3 — Phases 72–73 not visible |
AWS |
Action needed |
update-agent-runtime → python-tenants-v3 |
| D5 |
No quality benchmark on either platform — get_quality_metrics returns "no results" |
Both |
Low |
Phase 71 (COTS systemd timer); future AWS follow-up |
9. What the Gap Closures Proved
-
Phase 70 is the most impactful fix. It changed COTS from "broken-looking"
(0 docs, Unhealthy, 3 SKIPs) to "trustworthy" (220K docs, Healthy, real
findings). The unmasked Stale Embeddings WARN is a feature — the system is
now honest rather than silently hiding problems.
-
Phase 72 transformed Coverage Gap from useless to actionable. A [SKIP]
that told you nothing became a tri-language report: 107K Fortran nodes for
7.2K files, 8.6K Python for 41 files, 589 Shell for 114 files. Operators
can now see at a glance whether the graph covers the source tree.
-
Phase 73 eliminated the "three confusing numbers" problem. 108K
(health-check scope), 225K (tenant gw), 344K (all tenants) — each labeled,
each accessible, never disagreeing without explanation.
-
Graph traversal parity remains rock-solid. setuprad returns the same
55 direct neighbors on both Neptune and Neo4j. The knowledge graph is
functionally equivalent for code analysis regardless of backend.
10. Remaining Actions (Priority Order)
| # |
Action |
Spec |
Impact |
| 1 |
Update AWS runtime to python-tenants-v3 |
agentcore-image-rebuild-v3 |
Phases 72–73 live on AWS |
| 2 |
Run AWS doc refresh (44 stale + 14 never) |
aws-doc-refresh-ingest |
Resolves AWS Stale + Path WARNs |
| 3 |
Complete COTS re-ingest (44 pending units) |
cots-reingest-ralph-framework |
Resolves COTS Stale + fills v17 |
| 4 |
Install nightly benchmark timer |
nightly-rag-benchmark-harness |
Enables get_quality_metrics |
| 5 |
Retire legacy collections |
Future (post-framework) |
Cleanup; reduce confusion |
Report generated 2026-07-22 by Kiro AI, querying agentcore-mcp-rag (AWS) and
eib-mcp-gateway (COTS via SSH tunnel) simultaneously. Phases 70–73 confirmed
live on COTS. Compared against Platform-Parity-Check-AWS-vs-COTS-2026-07-15
baseline.