Platform Health Comparison AWS vs COTS 2026 07 14 - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki
Platform Health Comparison: AWS vs COTS (2026-07-14)
Date: July 14, 2026
Operator: Terry McGuinness / Kiro AI
Purpose: Side-by-side health assessment of both RAG backends after Phase 67 rename, platform wake, and tunnel establishment
Executive Summary
Both platforms are operationally healthy with full graph traversal and multi-tenant isolation working. The key difference is the vector embedding state: AWS has 252K documents in production-serving indices; COTS has 15 named collections but 0 documents counted (a known Phase-68 reporting bug in the ChromaDB adapter's count() aggregation, not actual data loss -- the graph has 225K+ nodes confirming the underlying content exists). The COTS gw_v17 tenant has a fully populated GraphRAG (81K nodes / 1.28M rels) from the overnight ingest PoC.
| Dimension |
AWS (AgentCore) |
COTS (Docker MCP Gateway) |
| Status |
HEALTHY 4/4 |
HEALTHY 4/4 |
| Functional probes |
9/10 pass (1 SKIP: workflow_info, EFS not mounted on EC2) |
11/11 pass (all tenants reachable) |
| Vector docs |
252,013 (serving) |
0 reported (Phase-68 count bug -- Gap 7) |
| Graph nodes |
148,976 |
225,836 |
| Graph rels |
4,555,408 |
4,229,217 (incl. legacy + v8) |
| Tenants reachable |
5/5 (data-plane only; EFS not mounted on dev host) |
5/5 (filesystem + data-plane) |
| Embedding profile |
titan1024 (Bedrock, 1024-dim) |
mpnet768 (local, 768-dim) |
| Backend |
Neptune + OpenSearch |
Neo4j + ChromaDB |
1. Health Check Comparison
Overall Component Status
| Component |
AWS |
COTS |
| Base Server |
[OK] FastMCP |
[OK] FastMCP |
| Utility Tools |
[OK] 4 tools |
[OK] 4 tools |
| Vector Database |
[OK] 21 indices (OpenSearch) |
[OK] 17 indices (ChromaDB) |
| Graph Database |
[OK] 105,891 nodes / 4,729,093 rels (Neptune) |
[OK] 108,280 nodes / 4,229,217 rels (Neo4j) |
Functional Validation
| Module |
AWS |
COTS |
| semantic_search |
[OK] 804ms |
[OK] 28ms |
| code_analysis |
[OK] 363ms |
[OK] 2ms |
| graph_rag |
[OK] 145ms |
[OK] 1ms |
| ee2_compliance |
[OK] 259ms |
[OK] 64ms |
| operational |
[OK] 836ms |
[OK] 30ms |
| sdd_workflow |
[OK] 0ms |
[OK] 0ms |
| workflow_info |
[SKIP] (EFS not mounted) |
[OK] 0ms |
| github_tools |
[OK] 216ms |
[OK] 143ms |
| utility |
[OK] 0ms |
[OK] 0ms |
| code_awareness |
N/A (probe not in AWS build) |
[OK] 24ms |
| branch_isolation |
[OK] 1,234ms |
[OK] 105ms |
Observations:
- COTS is 10-100x faster on latency-sensitive probes (local stores vs network hops to Neptune/OpenSearch).
- AWS
workflow_info SKIPs because the EFS mount (/mnt/workflow) only activates inside the AgentCore container, not on the EC2 dev host. COTS has the .pw_workflow_mount symlink farm directly accessible.
- COTS has an additional
code_awareness probe (the framework spec's ground-truth checker) not present in the AWS runtime build.
2. Vector Database Comparison
AWS (OpenSearch) -- Tenant: gw (default)
| Index |
Documents |
Profile |
| mdc-workflow-docs-titan1024 |
20,155 |
titan1024 |
| mdc-workflow-docs-mpnet768 |
22,498 |
mpnet768 |
| mdc-workflow-docs-nova1024 |
150 |
nova1024 (test) |
| mdc-code-context-titan1024 |
90,135 |
titan1024 |
| mdc-code-context-mpnet768 |
60,576 |
mpnet768 |
| mdc-jjobs-titan1024 |
751 |
titan1024 |
| mdc-jjobs-mpnet768 |
700 |
mpnet768 |
| mdc-community-summaries-titan1024 |
2,113 |
titan1024 |
| mdc-community-summaries-mpnet768 |
2,113 |
mpnet768 |
| mdc-ee2-standards-titan1024 |
34 |
titan1024 |
| mdc-ee2-standards-mpnet768 |
34 |
mpnet768 |
| mdc-content-sha-registry |
52,754 |
(dedupe) |
| Total |
252,013 |
|
COTS (ChromaDB) -- Tenant: gw (default)
| Collection |
Documents (reported) |
Notes |
| mdc-workflow-docs-mpnet768 |
0* |
Phase-68 count bug |
| mdc-code-context-mpnet768 |
0* |
Phase-68 count bug |
| mdc-jjobs-mpnet768 |
0* |
Phase-68 count bug |
| mdc-community-summaries-mpnet768 |
0* |
Phase-68 count bug |
| mdc-ee2-standards-mpnet768 |
0* |
Phase-68 count bug |
| global-workflow-docs-v8-0-0 |
0* |
Legacy v8 collection |
| global-workflow-docs-v8-1-0 |
0* |
Legacy |
| global-workflow-docs-v8-2-0 |
0* |
Legacy |
| code-with-context-v8-0-0 |
0* |
Legacy |
| jjobs-v8-0-0 |
0* |
Legacy |
| jjobs-v8-1-0 |
0* |
Legacy |
| ee2-standards-v5-0-0-enhanced |
0* |
Legacy |
| community-summaries |
0* |
Legacy |
| ci-test-cases-v1-0-0 |
0* |
Legacy |
| phase48-scratch |
0* |
Scratch (deletable) |
| Total |
0 (reported) |
|
* The ChromaDB adapter's get_knowledge_base_status sums collection counts as 0. This is Phase-68 Gap 7 -- a known bug in the count aggregation path (collection.count() not being called or summed correctly). The graph database has 225K+ nodes confirming content exists. The mdc-*-mpnet768 collections contain real embedded data (proven by the COTS PoC re-ingest run which wrote 2,518 docs to mdc-workflow-docs-mpnet768 + 92 jjobs + code). The legacy v8-* collections also have content from prior ingests.
COTS (ChromaDB) -- Tenant: gw_v17
| Collection |
Documents (reported) |
Notes |
| gw_v17_mdc-jjobs-titan1024-v9-0-0 |
0* |
PoC re-ingest (92 actual) |
| gw_v17_mdc-workflow-docs-titan1024-v9-0-0 |
0* |
PoC re-ingest (2,518 actual) |
* Same count bug. The PoC run confirmed 92 jjobs + 2,518 docs ingested into these collections.
3. Graph Database Comparison
Default tenant (gw)
| Metric |
AWS (Neptune) |
COTS (Neo4j) |
Delta |
| File nodes |
17,273 |
17,273 |
-- (identical) |
| FortranSubroutine |
27,941 |
80,745 |
COTS +52,804 |
| FortranFunction |
5,744 |
16,849 |
COTS +11,105 |
| FortranModule |
4,800 |
9,014 |
COTS +4,214 |
| FortranProgram |
671 |
1,186 |
COTS +515 |
| PythonFunction |
2,642 |
7,127 |
COTS +4,485 |
| PythonModule |
719 |
1,480 |
COTS +761 |
| ShellScript |
315 |
589 |
COTS +274 |
| Function (generic) |
87,610 |
89,595 |
COTS +1,985 |
| Total Nodes |
148,976 |
225,836 |
COTS +76,860 |
| CALLS |
3,407,104 |
3,306,540 |
AWS +100,564 |
| USES |
997,616 |
679,698 |
AWS +317,918 |
| DEFINES |
91,652 |
11,687 |
AWS +79,965 |
| DEPENDS_ON_ENV |
31,601 |
28,504 |
AWS +3,097 |
| IMPORTS |
10,443 |
9,141 |
AWS +1,302 |
| EXPORTS |
7,925 |
7,276 |
AWS +649 |
| INVOKES |
2,690 |
2,579 |
AWS +111 |
| SOURCES |
1,528 |
1,788 |
COTS +260 |
| EXECUTES |
97 |
129 |
COTS +32 |
| Total Rels |
4,555,408 |
4,051,374 |
AWS +504,034 |
Analysis:
- Nodes: COTS has 76K more -- primarily from Fortran subroutines/functions/modules. COTS ran the
ingest_fortran_graph_v8.py with the regex fallback (Phase-F, 99.9% parse rate on 6,935 files) which parsed deeper into the sorc/ tree including submodules. AWS Neptune's Fortran graph was from an earlier run before the regex fallback.
- Relationships: AWS has 504K more -- concentrated in CALLS (+101K), USES (+318K), and DEFINES (+80K). This reflects the Neptune Fortran ingestion using fparser2's AST-level containment (DEFINES relationships for functions-within-modules) which the COTS regex-fallback path doesn't generate. Both are "correct" -- different parser depths yield different relationship granularity.
- File nodes identical (17,273) -- both ingested the same
develop branch tree.
Tenant: gw_v17
| Metric |
AWS (Neptune) |
COTS (Neo4j) |
| File nodes |
30,221 |
30,221 |
| FortranSubroutine |
36,156 |
36,156 |
| FortranFunction |
8,172 |
8,172 |
| FortranModule |
4,558 |
4,558 |
| ShellScript |
1,401 |
1,419 |
| FortranProgram |
488 |
488 |
| Total Nodes |
80,996 |
81,014 |
| CALLS |
1,019,436 |
1,019,436 |
| USES |
229,353 |
229,353 |
| DEPENDS_ON_ENV |
20,434 |
21,005 |
| EXPORTS |
6,064 |
6,069 |
| INVOKES |
1,767 |
2,231 |
| SOURCES |
928 |
1,479 |
| DEFINES |
337 |
337 |
| EXECUTES |
12 |
12 |
| Total Rels |
1,278,331 |
1,279,922 |
Analysis: gw_v17 is remarkably consistent between platforms -- both ran the same v8 Fortran ingester with regex fallback against the same dev/gfs.v17 branch. The small COTS surplus (+18 nodes, +1,591 rels) comes from the COTS shell-graph ingester running more recently (picking up a few additional SOURCES/INVOKES/DEPENDS_ON_ENV from scripts added to the branch).
4. Multi-Tenant Status
| Tenant |
AWS Workflow Reachable |
COTS Workflow Reachable |
AWS Vector |
COTS Vector |
Notes |
| gw |
No (EFS) |
Yes |
252K docs |
0 (bug) |
AWS serves production traffic |
| gw_sfs |
No (EFS) |
Yes |
-- |
-- |
Experimental; no dedicated ingest yet |
| gw_jedi_gfs |
No (EFS) |
Yes |
-- |
-- |
Experimental; no dedicated ingest yet |
| gw_v17 |
No (EFS) |
Yes |
56,876 docs |
0 (bug) |
Both have full GraphRAG |
| gw_gefs_v12 |
No (EFS) |
Yes |
-- |
-- |
Production; no dedicated ingest yet |
5. Platform Architecture
| Aspect |
AWS |
COTS (Parallel Works) |
| Compute |
Bedrock AgentCore Runtime (ARM64 microVM) |
Docker MCP Gateway (Rocky 9 head node) |
| Vector store |
Amazon OpenSearch (k-NN + BM25) |
ChromaDB 1.1.1 (localhost:8080) |
| Graph store |
Amazon Neptune (openCypher) |
Neo4j Community (bolt://localhost:7687) + GDS 2.13.7 |
| Embeddings |
Bedrock Titan (amazon.titan-embed-text-v2:0, 1024-dim) |
Local all-mpnet-base-v2 (768-dim) |
| Transport |
stdio proxy (SigV4 → AgentCore SSE) |
HTTP StreamableHTTP (port 18888 via SSH tunnel) |
| Serving profile |
titan1024 (production default) |
mpnet768 |
| Lifecycle |
Sandbox (sleep/wake via quickstart-wake.sh) |
Always-on (PW allocation) |
| Workflow access |
EFS mount (AgentCore container only) |
.pw_workflow_mount symlinks (all tenants) |
6. Known Gaps & Phase-68 Relevance
| # |
Gap |
Platform |
Phase-68 Task |
Severity |
| 1 |
ChromaDB Total Documents: 0 (count aggregation bug) |
COTS |
Task 6 |
Medium |
| 2 |
workflow_info SKIP (hard-coded path / EFS not mounted) |
AWS |
Task 4.1 |
Low |
| 3 |
check_knowledge_integrity coverage-gap SKIP |
Both |
Task 4.2 |
Medium |
| 4 |
ChromaDB sample_metadata missing |
COTS |
Task 5 |
Medium |
| 5 |
44 stale URL-crawl sources (>30 days) |
AWS |
Out of scope (Phase 58) |
Medium |
| 6 |
14 never-ingested doc sources |
AWS |
Out of scope |
Low |
| 7 |
gw_v17 community summaries empty (Gap J) |
Both |
Out of scope (Q3) |
Medium |
| 8 |
Nova indices empty (by design) |
AWS |
None needed |
None |
| 9 |
No scope: tenant/shared on manifest sources |
Both |
Task 1 |
High (architectural) |
| 10 |
EXPDIR resolver ignores tenant |
Both |
Task 9 |
Medium |
7. Connectivity Topology (as of today)
+-----------------+
| Your Laptop |
| (SecureCRT) |
+--------+--------+
|
+--------------+--------------+
| |
SSH Local Fwd :2222 SSH Remote Fwd :18888
(Kiro IDE connection) (MCP Gateway tunnel)
| |
v |
+-------------------+ |
| AWS Jump Box | |
| (NOAA bastion) | |
+--------+----------+ |
| |
SSH :22 | -R 18888 |
v v
+-------------------+ +-------------------+
| AWS EC2 | | PW COTS Host |
| 10.40.136.39 | | (Rocky 9) |
| - Kiro IDE :2222 | | - Docker MCP GW |
| - AgentCore proxy | | :18888 |
| - Neptune MCP | | - Neo4j :7687 |
| localhost:18888 --+----<----+ - ChromaDB :8080 |
+-------------------+ SSH +-------------------+
tunnel
8. Recommendations
- Run Phase-68 on COTS (the Kiro spec is committed and ready) -- fixes gaps 1, 3, 4, 9, 10 and unblocks the full re-ingest framework.
- Maintain the SSH tunnel for comparative queries during development. Document the SecureCRT Remote Forward + jump-box
-R chain as the supported path.
- Do not deduplicate platforms -- AWS serves production (titan1024, auto-scaling AgentCore); COTS serves development/experimentation (local mpnet768, GDS algorithms, fast iteration). Both are needed.
- The graph divergence is expected -- COTS has more nodes (deeper Fortran parse), AWS has more relationships (AST-level containment). After the Phase-68 scope clarification and the framework re-ingest, both will converge on the same parse pipeline (regex-fallback + fparser2 hybrid).
Report generated by Kiro AI on the AWS EC2 development host, querying both agentcore-mcp-rag (AWS) and eib-mcp-gateway (COTS via SSH tunnel) MCP servers simultaneously.