MCP RAG Platform June July 2026 Milestone Report - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki
MDC MCP-RAG Platform โ June / July 2026 Milestone Report
Period: June 1 โ July 6, 2026 (115 commits, branch develop_aws_startpoint)
Server: MDC MCP/RAG v1.0.0 ยท 53 tools ยท 9 active modules ยท 5 tenants
Status at close of period: Gateway healthy โ 11/11 functional tests pass, 4/4 components healthy
Executive Summary
Over five weeks the MDC MCP-RAG platform completed its transition from a single-tenant, AWS-only research prototype to a production-grade, five-tenant, dual-backend (COTS/AWS) system running natively on Parallel Works and exposing a 53-tool MCP endpoint via the Docker MCP Gateway (port 18888, Streamable HTTP). Ten numbered phases and six multi-phase gap-closure sprints were completed. The period ended with the retirement of the legacy static Node.js container and full functional parity between the Python stdio server and the gateway-spawned Python container.
Milestone 1 โ Multi-Tenant Foundation (June 1โ5)
Phases: Gap A (tenant-id tool exposure), provisioning v4.1.0 multi-user
| Item | Detail |
|---|---|
| Tenant-id wired to 24 tools | All semantic search, code analysis, graph-RAG, EE2, and operational tools now accept tenant_id; isolates query results per branch |
| Cross-tenant leakage fix | Added :Label qualifiers to all Cypher MATCH patterns preventing gw data bleeding into gw_v17 or gw_sfs results |
| Five tenants registered | gw (develop), gw_sfs (dev/sfs), gw_jedi_gfs (dev/jedi-gfs), gw_v17 (dev/gfs.v17), gw_gefs_v12 (release/gefs_v12) |
| Multi-user provisioning | 00-users.sh extended to 5 workflow-developer accounts; .bashrc template, uv/uvx, git-branch prompt, scratch workspace per-user |
| GitHub token from Secrets Manager | AWS-standard startup path; eliminates plaintext PAT in env vars |
| New RAG doc sources | ECMWF Atlas + JEDI Academy 2021-06/10 added โ 58 sources, 56 enabled |
Milestone 2 โ Gap-Closure Sprint: Graph Porting & Tenant Isolation (June 3โ11)
Phases: Gap A close, Gap B (graph port), Gaps D, E, F
| Gap | Fix | Release |
|---|---|---|
| Gap A | Tenant-id tool exposure deployed (v22); all 24 tools scoped | โ |
| Gap D | Bracket-aware tenant label rewriter for complex Cypher queries | 8.31.0 |
| Gap E | Tenant-scope added to label-less graph queries (prevents global fan-out) | 8.32.0 |
| Gap F | Fortran parse fallback (regex when AST fails) + parallel parse runner + streaming OOM fix | 8.33.0 / 8.34.0 |
| Gap B | Graph port series โ Shell ops, workflow structure, and Fortran AST parsers ported to Python pipeline; 9 tasks completed | โ |
| Gap G | Bounded graph traversal deployed (v32); gap closed | โ |
| Gap I | v17 knn-vector reindex (float โ knn_vector field type); gap resolved | โ |
| OpenSearch Bug 1 | resolve_index called before tenant-prefix applied in adapter query |
8.36.2 |
| OpenSearch Bug 2 | _render_vector_status_block not tenant-scoped in knowledge-base status |
8.36.2 |
| Health-check Bug 1 | _parse_iso_ts UTC fallback + skip-count reconciliation (R3.5) |
8.36.1 |
| Health-check Bug 2 | _smoke_workflow_info SKIPs (non-fatal) when workflow mount missing |
8.36.1 |
| Graceful missing-index | 4 tools now return [INFO] Skip_Block instead of error when index absent |
8.36.3 |
Experiment directories imported: 17 clean EXPDIR entries (develop) + 9 from dev/gfs.v17 for workflow-structure ingestion.
Milestone 3 โ AWS Cost Control IaC & Data Portability (June 15)
Phases: NIH Sandbox Cost Control, Cross-Platform Data Persistence
| Item | Detail |
|---|---|
| CDK sleep/wake IaC โ Waves 0โ6 | Full CDK app covering EC2, Neptune, OpenSearch, AgentCore, and NAT tier hibernation/wake; Wave 3 state machine + CLI orchestrator composes all tiers |
| Drift detection | Wave 2 Group C: detects resource drift from expected sleep state; wake probe verifies services healthy after resume |
| Quickstart hibernate/wake scripts | Operator-facing CLI scripts for end-of-day sleep and morning wake of the entire NIH sandbox |
| Symmetric export/restore adapters | Bidirectional AWS โ COTS pipeline: Neptune โ Parquet โ Neo4j and ChromaDB โ Parquet โ OpenSearch (and reverse); 4-mode operation |
| Portable export live artifacts | neptune_reader fixed for label-family streaming; sync_to_pw.sh + s3_cross_account_copy.py for PW bucket sync; 20 ingestion run reports (May 28 โ Jun 11) |
| IAM policy tightened | CDK orchestrator IAM policy reviewed and least-privileged per pre-deploy review |
Milestone 4 โ Python MCP Server Integration on Parallel Works (June 24)
Phases: Python MCP PW legacy-backend integration, Phase 60 (code awareness parity + branch validation)
| Item | Detail |
|---|---|
| COTS backend integration | mcp_server_python running natively on PW Rocky 9 against local ChromaDB + Neo4j via DB_BACKEND=cots; all probes backend-agnostic |
| Backend-agnostic graph health probe | Neo4j per-label node count replaces AWS-specific OpenSearch probe โ works on both backends |
| Code awareness parity | code_awareness and branch_isolation functional test modules added; both pass (411 ms, 248 ms) |
| Phase 61: Configurable workflow mount | MCP_WORKFLOW_MOUNT env var + .pw_workflow_mount/ symlink farm; all 5 tenant workflow_root paths resolve on PW; get_workflow_structure/list_job_scripts/find_env_dependencies fully operational |
| Docker Gateway auth fix | Authorization token header wired correctly in streaming transport |
| SDD steering additions | Commit message standards + tool-count reconciliation directives added to .kiro/steering/ |
Milestone 5 โ CI Error Log Distillation Tool (June 26)
Phase: Phase 62
| Item | Detail |
|---|---|
extract_ci_error_signal tool |
New MCP tool distils multi-MB CI log files to an 8 KB LLM-consumable footprint; applies noise filtering (bash env dumps, module chatter) and captures failure signals with tail tracebacks + exit codes |
ErrorRecord schema |
Standardized payload schema for structured failure extraction |
| Ordered failure taxonomy | Classifier applies priority-ordered failure taxonomy before signal capture |
| Reporting directive | .kiro/steering/13-ci-error-reporting-policy.md mandates MCP tooling for all CI log analysis |
| Module registration fix | error_analysis module registered in KNOWN_MODULES (post-Phase 62 hotfix) |
Milestone 6 โ Backend Label Rename: legacy โ cots (July 2)
Phase: Phase 63a
| Item | Detail |
|---|---|
DB_BACKEND=cots canonical |
Renames the on-prem backend value from legacy to cots (Commercial Off-The-Shelf: ChromaDB + Neo4j); accurately names the stack at the same conceptual level as aws |
| Deprecation shim | DB_BACKEND=legacy still accepted through one release cycle with a single [WARN] per process; auto-maps to cots |
| Removal scheduled | Phase 64 โ is_legacy() alias and legacy shim removed |
| Scope | Python server, Node scripts, .vscode/mcp.json example, unit tests โ 15+ files updated |
Milestone 7 โ Docker MCP Gateway Python Parity (July 3)
Phase: Phase 63b โ 11/11 functional tests passing
| Item | Detail |
|---|---|
| New gateway image | eib-mcp-rag-python:latest (x86_64 Python build) replaces eib-mcp-rag:latest (Node.js) in the catalog |
| Tool parity | 53 tools / 9 modules / 5 tenants โ matches stdio server exactly |
| COTS dependencies baked | chromadb==1.3.4, neo4j==5.25.0, sentence-transformers==5.1.2, torch==2.9.0+cpu (CPU index); Spack-pinned |
| Offline mpnet768 embeddings | HF_HUB_OFFLINE=1; 768-dim mpnet encoder loads from host HF cache mounted read-only at /app/.hf_cache |
| Relative symlinks | .pw_workflow_mount uses realpath --relative-to so links resolve identically on host and inside container |
GitHub token via --secrets |
GITHUB_TOKEN wired through docker-mcp secrets: catalog block + mcp-gateway-launch.sh wrapper sourcing ~/.config/eib-mcp/secrets.env onto tmpfs |
| Health verification | 4/4 components healthy; 15 collections / 220,538 docs; 108,280 nodes / 4,220,211 relationships; all 5 tenants workflow_root reachable = yes |
ExecStopPost shell-pipe note |
Harmless journal error (unknown shorthand flag: -r) on service stop; wrap in /bin/sh -c '...' deferred to Phase 63x cleanup |
Milestone 8 โ Retire Static Node Container (July 3)
Phase: Phase 63c
| Item | Detail |
|---|---|
mcp-rag.service retired |
Stopped, disabled; no longer a prerequisite of mcp-gateway.service |
| 8 GB / 4 CPU freed | eib-mcp-rag-static container (Node.js, single-tenant, 51 tools) no longer running idle |
| Dependency rewrite | mcp-gateway.service [Unit] Requires= rewritten from mcp-rag.service โ docker.service (drop-ins cannot reset dependency lists; base unit edited directly; backup at .bak-phase63c) |
| Cron resurrection gap | /etc/cron.d/mcp-health (*/5 * * * *) was silently restarting mcp-rag.service after disable; Rocky 9 crond reads dotted filenames so rename to .disabled was insufficient โ file deleted |
| In-tree deprecation | 12-static-mode-gateway.sh short-circuits exit 0; SETUP/cron.d/mcp-health, SETUP/bin/health-check.sh, SETUP/bin/deploy-static-gateway.sh all marked DEPRECATED with rollback guards |
| Rollback preserved | Node image eib-mcp-rag:latest (id cc69b32e3c67) retained locally; one-line catalog revert restores the prior topology |
Milestone 9 โ ARM Dev-Host Migration Spec + Kiro CLI Fix (July 2)
Phase: Phase 65 (spec; implementation deferred to week of Jul 6)
| Item | Detail |
|---|---|
| Phase 65 IaC spec | DevHostStack (CDK) to replace console-created c6g.xlarge with IaC-managed r7g.2xlarge ARM AL2023 AMI; EFS-mounted, wired to existing Neptune/OpenSearch/Bedrock stacks |
| Kiro CLI musl fix | AL2023 glibc 2.34 incompatible with glibc-2.38/2.39 kiro-cli update build; SETUP/update-kiro-cli-musl.sh pulls the statically-linked musl build instead |
| Dev tunnel utility | Helper script for managing VS Code dev tunnel connections added |
Platform State at 2026-07-06
| Metric | Value |
|---|---|
| Tools | 53 (9 active modules) |
| Tenants | 5 (default: gw) |
| ChromaDB collections | 15 |
| Total vector documents | 220,538 |
| Neo4j nodes | 225,836 |
| Neo4j relationships | 4,049,454 |
| RAG doc sources | 58 (56 enabled) |
| Functional test result | 11/11 pass |
| Gateway uptime | mcp-gateway.service active + enabled; Restart=always |
| SDD sessions | 65 workflows, 48 sessions (34 completed, 5 abandoned) |
Open Gaps (carried into July)
| Gap | Description | Priority |
|---|---|---|
| Non-default tenant ingestion | gw_sfs, gw_jedi_gfs, gw_gefs_v12 have zero vectors + zero graph; gw_v17 has graph but no vectors |
High |
check_knowledge_integrity scoping |
Coverage-gap and orphaned-node checks are not tenant-scoped; false-positive "all passed" on empty tenants | Medium |
| Quality benchmark | quality_metrics.jsonl absent; get_quality_metrics returns no data |
Medium |
Phase 63c ExecStopPost |
Shell pipe | xargs -r produces harmless journal error on gateway stop |
Low |
| Phase 64 cleanup | Remove DB_BACKEND=legacy shim and is_legacy() alias |
Low |
Generated 2026-07-06 from git log --since=2026-06-01 develop_aws_startpoint (115 commits) and live Docker MCP Gateway health tools.