Platform Parity Check AWS vs COTS 2026 07 22 - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

Platform Parity Check: AWS vs COTS — Post-Gap-Closure Assessment (2026-07-22)

Date: July 22, 2026 Operator: Terry McGuinness / Kiro AI Purpose: Full-scope parity check with Phases 70–73 gap closures live on COTS (container restarted, new code active). AWS runtime still on python-tenants-v11 (pre-Phase-70). Confirms which gaps closed, surfaces new deeper findings, and documents remaining actions.


Executive Summary

The COTS gap closures (Phases 70–73) are proven working. The Docker MCP Gateway went from "3 integrity SKIPs + false Unhealthy + unlabeled counts" to a trustworthy, honest, multi-language coverage report with real document counts and self-describing scope annotations. The most impactful single fix (Phase 70: ChromaDB count_documents + sample_metadata) unmasked a real Stale Embeddings finding (50/50 docs >30 days) that was previously hidden by the SKIP — the system is now honest rather than silently broken. AWS remains healthy but still runs the old image (Phases 72–73 not yet visible there).

Dimension AWS (AgentCore) COTS (Docker MCP Gateway) Parity
Overall health HEALTHY 4/4 HEALTHY 4/4 MATCH
Functional probes 9/10 pass (1 SKIP: workflow_info/EFS) 11/11 pass COTS leads
Vector docs (gw) 252,013 220,538 AWS leads (+31K; dual-profile)
Graph nodes (gw) 148,976 225,836 COTS leads (+77K; deeper Fortran)
Graph rels (gw) 4,555,408 4,051,374 AWS leads (+504K; AST containment)
Whole-graph count (not available — old image) 344,604 (Phase 73) COTS only
Integrity: Path [WARN] 2/34 [OK] 0/50 COTS leads (Phase 70)
Integrity: Stale [WARN] 12/12 [WARN] 50/50 Both stale (real finding)
Integrity: Coverage [SKIP] stale path [OK] 107K Fortran + Python + Shell COTS leads (Phase 72)
Integrity: Orphans [OK] 0/20 [OK] 0/20 MATCH
setuprad traversal 55 neighbors (degree 193) 55 neighbors (degree 174) MATCH

1. Gap Closures Confirmed Live on COTS

Phase Gap (before) Result (after) Impact
70 Total Documents: 0 [ERROR] Unhealthy Total Documents: 220,538 [OK] Healthy Real doc counts; false unhealthy eliminated
70 [SKIP] adapter does not expose a metadata sampler (×2) Path: [OK] 0/50; Stale: [WARN] 50/50 Integrity checks execute; real findings unmasked
72 [SKIP] no Fortran files found in ... [OK] 107,794 Fortran nodes for 7,242 files Multi-language coverage: Fortran + Python (8,607/41) + Shell (589/114)
73 108280 nodes (unlabeled) 108,280 nodes (health-check scope) Self-describing scope annotation
73 No whole-graph visibility Total Nodes (all tenants, all labels): 344,604 Full graph via all_tenants=True
73 Total Nodes: 225836 (unlabeled) Total Nodes (tenant gw): 225,836 Tenant-scoped, self-describing
70 v17 Total Documents: 0 [ERROR] Total Documents: 2,610 [OK] Healthy PoC re-ingest data (92 jjobs + 2,518 docs) now visible

2. Health Check Comparison

Component Status

Component AWS COTS
Base Server [OK] FastMCP [OK] FastMCP
Utility Tools [OK] 4 tools [OK] 4 tools
Vector Database [OK] 21 indices [OK] 17 indices
Graph Database [OK] 105,891 nodes / 4,729,093 rels [OK] 108,280 nodes (health-check scope) / 4,229,217 rels

Functional Validation

Module AWS COTS
semantic_search [OK] 175ms [OK] 102ms
code_analysis [OK] 49ms [OK] 3ms
graph_rag [OK] 46ms [OK] 2ms
ee2_compliance [OK] 161ms [OK] 93ms
operational [OK] 188ms [OK] 89ms
sdd_workflow [OK] 0ms [OK] 0ms
workflow_info [SKIP] (EFS not mounted) [OK] 0ms
github_tools [OK] 200ms [OK] 104ms
utility [OK] 0ms [OK] 0ms
code_awareness N/A [OK] 28ms
branch_isolation [OK] 472ms [OK] 67ms

3. Knowledge Base Status — Default Tenant (gw)

Vector Database

Metric AWS (OpenSearch) COTS (ChromaDB)
Collections 16 15
Total Documents 252,013 220,538
Status [OK] Healthy [OK] Healthy
Serving profile titan1024 (1024-dim) mpnet768 (768-dim)

COTS collection breakdown (Phase 70 now reporting real counts):

Collection Documents Notes
mdc-workflow-docs-mpnet768 22,498 Serving docs
mdc-code-context-mpnet768 60,576 Serving code
mdc-jjobs-mpnet768 700 Serving jjobs
mdc-ee2-standards-mpnet768 34 Serving EE2
mdc-community-summaries-mpnet768 2,113 Serving community
global-workflow-docs-v8-2-0 23,624 Legacy (v8.2)
global-workflow-docs-v8-1-0 20,511 Legacy (v8.1)
global-workflow-docs-v8-0-0 22,498 Legacy (v8.0)
code-with-context-v8-0-0 60,574 Legacy code
jjobs-v8-0-0 700 Legacy jjobs
jjobs-v8-1-0 859 Legacy jjobs v8.1
ee2-standards-v5-0-0-enhanced 34 Legacy EE2
community-summaries 2,113 Legacy community
ci-test-cases-v1-0-0 74 CI test cases
phase48-scratch 3,630 Scratch (deletable)

Graph Database

Metric AWS (Neptune) COTS (Neo4j)
Total Nodes 148,976 225,836 (tenant gw) / 344,604 (all tenants)
Total Rels 4,555,408 4,051,374
File 17,273 17,273
FortranSubroutine 27,941 80,745
FortranFunction 5,744 16,849
ShellScript 315 589

4. Integrity Check Comparison

Check AWS COTS Analysis
Path Consistency [WARN] 2/34 [OK] 0/50 COTS clean; AWS has 2 old-prefix docs (need re-ingest)
Orphaned Graph Nodes [OK] 0/20 [OK] 0/20 Both clean
Stale Embeddings [WARN] 12/12 [WARN] 50/50 Both stale — AWS 30+ days; COTS ~50+ days. Real finding on both; need re-ingest
Coverage Gap [SKIP] stale path [OK] 107,794 Fortran / 8,607 Python / 589 Shell Phase 72 transformed this from useless to actionable

5. Knowledge Base Status — Tenant gw_v17

Metric AWS (Neptune) COTS (Neo4j) Delta
File 30,221 30,221 MATCH
FortranSubroutine 36,156 36,156 MATCH
FortranFunction 8,172 8,172 MATCH
FortranModule 4,558 4,558 MATCH
ShellScript 1,401 1,419 COTS +18
Total Nodes 80,996 81,014 COTS +18
Total Rels 1,278,331 1,279,922 COTS +1,591
Vector docs 56,876 2,610 AWS leads (full ingest vs partial PoC)

6. Graph Traversal Parity: setuprad

Metric AWS (Neptune) COTS (Neo4j)
Measured degree 193 174
Neighbors returned 55 55
Function overlap ~98% identical ~98% identical

Both platforms return the same setuprad call-graph: qc_irsnd, qc_amsua, qc_mhs, qc_atms, init_crtm, call_crtm, destroy_crtm, calc_clw, ret_amsua, nc_diag_init, nc_diag_write, setup_sst_retrieval, get_radiag, comp_fact10, dtime_setup, die, stop2, new, etc.


7. Health Trend (COTS — 10 snapshots)

Date Neo4j Nodes Neo4j Rels ChromaDB Docs Collections Doc Drift
2026-07-02 108,280 4,220,211 220,538 15 0
2026-07-09 22:07 108,280 4,229,217 223,148 17 +2,610
2026-07-10 – 2026-07-20 108,280 4,229,217 223,148 17 0

Trend: Graph nodes rock-stable (108,280 across 10 snapshots). Rels +9,006 (Jul 9 v17 shell-graph run). Docs +2,610 (Jul 9 v17 PoC). No anomalies.


8. Deeper Findings (New Gaps Surfaced)

# Finding Platform Severity Resolution
D1 Stale Embeddings (50/50 on COTS; 12/12 on AWS) — now unmasked by Phase 70. All sampled docs >30 days old on both platforms. Both Medium Full re-ingest on both (framework spec for COTS; aws-doc-refresh-ingest for AWS)
D2 COTS v17 vector partial (2,610 vs AWS 56,876) — PoC only partially completed COTS Medium Framework spec's 44 pending units
D3 10 legacy collections on COTS still serving alongside mdc-* collections COTS Low Post-framework cutover + retirement
D4 AWS runtime on python-tenants-v11 not v3 — Phases 72–73 not visible AWS Action needed update-agent-runtimepython-tenants-v3
D5 No quality benchmark on either platformget_quality_metrics returns "no results" Both Low Phase 71 (COTS systemd timer); future AWS follow-up

9. What the Gap Closures Proved

  1. Phase 70 is the most impactful fix. It changed COTS from "broken-looking" (0 docs, Unhealthy, 3 SKIPs) to "trustworthy" (220K docs, Healthy, real findings). The unmasked Stale Embeddings WARN is a feature — the system is now honest rather than silently hiding problems.

  2. Phase 72 transformed Coverage Gap from useless to actionable. A [SKIP] that told you nothing became a tri-language report: 107K Fortran nodes for 7.2K files, 8.6K Python for 41 files, 589 Shell for 114 files. Operators can now see at a glance whether the graph covers the source tree.

  3. Phase 73 eliminated the "three confusing numbers" problem. 108K (health-check scope), 225K (tenant gw), 344K (all tenants) — each labeled, each accessible, never disagreeing without explanation.

  4. Graph traversal parity remains rock-solid. setuprad returns the same 55 direct neighbors on both Neptune and Neo4j. The knowledge graph is functionally equivalent for code analysis regardless of backend.


10. Remaining Actions (Priority Order)

# Action Spec Impact
1 Update AWS runtime to python-tenants-v3 agentcore-image-rebuild-v3 Phases 72–73 live on AWS
2 Run AWS doc refresh (44 stale + 14 never) aws-doc-refresh-ingest Resolves AWS Stale + Path WARNs
3 Complete COTS re-ingest (44 pending units) cots-reingest-ralph-framework Resolves COTS Stale + fills v17
4 Install nightly benchmark timer nightly-rag-benchmark-harness Enables get_quality_metrics
5 Retire legacy collections Future (post-framework) Cleanup; reduce confusion

Report generated 2026-07-22 by Kiro AI, querying agentcore-mcp-rag (AWS) and eib-mcp-gateway (COTS via SSH tunnel) simultaneously. Phases 70–73 confirmed live on COTS. Compared against Platform-Parity-Check-AWS-vs-COTS-2026-07-15 baseline.