Date: 2026-06-10 Server: agentcore-mcp-rag (AWS Bedrock AgentCore Runtime, ARN arn:aws:bedrock-agentcore:us-east-1:<ACCOUNT_ID>:runtime/<PYTHON_RUNTIME_ID>) Method: Direct agentcore-mcp-rag MCP utility / GraphRAG calls, cross-checked against the deployed source tree at /mnt/mdc-mcp-rag/eib-mcp-rag-server.

Companion to agentcore-mcp-rag Tool Usage Analysis (v17 GEMPAK) — that page rates task-level usefulness of the tools; this page is the operational health of the deployment plus root-cause analysis of issues uncovered.

Bottom line: the deployment is up and serving graph + vector queries on the gw baseline; non-gw tenants are graph-only, and there are three concrete bugs (timezone, ingestion-writes-zero-docs, traversal timeout) that need fixes.

MDC MCP/RAG Server v1.0.0
Total Tools: 52   Active Modules: 9 of 9   Tenants: 5 (default: gw)

Active Modules:
  semantic_search, code_analysis, graph_rag, ee2_compliance, operational,
  sdd_workflow, workflow_info, github_tools, utility

Capabilities:
  Data Access: connected
  Vector Search: available
  Graph Queries: available
  Utility Tools: always-on (4 tools registered)

Overall Status: HEALTHY (4/4 components healthy)
[OK] Base Server     : FastMCP running
[OK] Utility Tools   : 4 utility tools registered
[OK] Vector Database : 5 indices
[OK] Graph Database  : 105891 nodes, 4729092 relationships

Tenants (5):
  gw           develop          production    no  /mnt/workflow/develop      (not mounted)
  gw_sfs       dev/sfs          experimental  no  /mnt/workflow/dev-sfs      (not mounted)
  gw_jedi_gfs  dev/jedi-gfs     experimental  no  /mnt/workflow/dev-jedi-gfs (not mounted)
  gw_v17       dev/gfs.v17      staging       no  /mnt/workflow/dev-v17      (not mounted)
  gw_gefs_v12  release/gefs_v12 production    no  /mnt/workflow/gefs-v12     (not mounted)

Default tenant: gw (resolved from catalog.defaults.tenant_id)
Workflow Filesystem mount: /mnt/workflow (NOT mounted)

The base-health node count (105,891 / 4,729,092) and the get_knowledge_base_status count below (148,976 / 4,555,407) disagree — they measure different things (live Neptune count vs cached aggregate) and both are reported as healthy. Worth tracking, not blocking.

Vector — 16 OpenSearch collections, 252,081 docs, all healthy:

Graph (Neptune) — 148,976 nodes / 4,555,407 relationships:

Edges:  CALLS 3,407,104   USES 997,616   DEFINES 91,652
        DEPENDS_ON_ENV 31,601   IMPORTS 10,443   EXPORTS 7,925
        DEPENDS_ON 4,752   INVOKES 2,690   SOURCES 1,528   EXECUTES 96
Nodes:  Function 87,610   FortranSubroutine 27,941   File 17,273
        FortranFunction 5,744   FortranModule 4,800   PythonFunction 2,642
        PythonModule 719   FortranProgram 671   Module 980   ShellScript 315

Vector — identical totals to gw (the API returns the global vector store view, not a tenant-scoped view — see §3.1).

Graph (gw_v17) — 80,996 nodes / 1,278,330 relationships:

Edges:  CALLS 1,019,436   USES 229,353   DEPENDS_ON_ENV 20,434
        EXPORTS 6,064   INVOKES 1,767   SOURCES 928
        DEFINES 337   EXECUTES 11
Nodes:  FortranSubroutine 36,156   File 30,221   FortranFunction 8,172
        FortranModule 4,558   ShellScript 1,401   FortranProgram 488

Notable absences in the gw_v17 graph:

• No Python labels (PythonFunction, PythonModule) — Python ingestion has not run on this tenant.
• DEFINES collapsed from 91,652 → 337 — the relationship that pairs every Fortran subroutine/function with its containing module. On gw_v17 only ~0.4% of the expected DEFINES exist; subroutines are present as nodes but largely orphaned from their module containers.
• No IMPORTS / DEPENDS_ON edges — the dependency-graph tools that rely on these will return sparse results on gw_v17.

Version: 6.0 Phase 31     Status: Operational
Available Workflows: 0    Total Sessions: 0
Active Session: none
[OK] start_sdd_session, record_sdd_step, get_sdd_session,
     complete_sdd_session, get_sdd_execution_history
[..] ISD approval (dormant — reserved for Phase 4C USD)
[..] WorkflowExecutor (filesystem-backed; degraded on AgentCore microVM)
[OK] SpecificationParser, SelfModificationEngine

get_health_trend  → "No health history found."
get_quality_metrics → "No benchmark results found at
                       /app/sdd_framework/execution_state/quality_metrics.jsonl"

Tested against gw, gw_sfs, gw_v17. All three failed identically:

[ERROR] can't subtract offset-naive and offset-aware datetimes

Server-wide bug, not tenant-specific.

So the tool degrades gracefully (good), but the default depth is unusably high for non-trivial roots, and even depth 1 takes ~15 s.

Same root cause — the gw_v17_* indices don't exist — but the error path is inconsistent (one tool says "no results", three return raw OpenSearch 404s).

get_knowledge_base_status reports the physical OpenSearch domain (all collections), not a tenant-prefixed view. Per mcp_server_python/src/data/opensearch_adapter.py:

def resolve_tenant_index(self, collection: str, tenant) -> str:
    """Apply tenant.index_prefix to a logical collection name."""
    if not tenant.index_prefix:
        return collection
    return f"{tenant.index_prefix}{collection}"

The prefix is applied at query time by individual search tools, so a tenant that has no prefixed indices yet (gw_v17, gw_sfs, etc.) will hit NotFoundException on real searches even though kb_status shows a "healthy" domain. This is misleading and is the strongest cause of false positives in operational monitoring.

The latest ingestion report at mcp_server_python/scripts/ingestion_reports/gw_v17_20260610T232209.json:

{
  "tenant_id": "gw_v17",  "mode": "bridge",
  "elapsed_seconds": 0.4,  "total_files_processed": 61,
  "documents_created": {},
  "embedding_calls": { "bedrock_invocations": 0, "estimated_tokens": 0 },
  "graph": { "nodes_created_by_label": {}, "relationships_created": 16 },
  "drift_flags": [
    "dedupe_efficiency_pct", "documents_created_total", "estimated_tokens"
  ]
}

A second report from earlier the same day in mode: "full" (3-hour run, 6,935 files) shows:

{
  "documents_created": {},   // ZERO documents created
  "embedding_calls": { "bedrock_invocations": 0 },  // Bedrock never invoked
  "graph": {
    "nodes_created_by_label": {
      "GW_V17_FortranModule": 4869, "GW_V17_FortranSubroutine": 31543,
      "GW_V17_FortranFunction": 8200, "GW_V17_FortranProgram": 543
    },
    "relationships_created": 297712
  }
}

The harness itself raises drift flags (dedupe_efficiency_pct, documents_created_total, estimated_tokens) — it knows nothing was written — but the run continues and exits 0. Hence:

• OpenSearch side: no gw_v17_* indices created → all per-tenant vector searches return NotFoundException.
• Neptune side: only the Fortran walker fires (32K subroutines / 8K functions ingested as nodes); the Python and Shell graph walkers either didn't run or didn't write DEFINES edges.

This explains every Gap B symptom we have observed. It is an active ingestion-pipeline bug, not just "porting in progress".

In mcp_server_python/src/tools/semantic_search.py:

# now is timezone-aware
now = datetime.now(timezone.utc)

# _parse_iso_ts returns whatever fromisoformat gives us:
def _parse_iso_ts(raw):
    if raw.endswith("Z"):
        raw = raw[:-1] + "+00:00"
    return datetime.fromisoformat(raw)   # NAIVE if input has no offset

# later:
if (now - mod_time).days > STALE_EMBEDDING_DAYS:   # CRASHES

If any sampled document's lastModified/ingestedAt is a bare ISO string (no Z, no +HH:MM), _parse_iso_ts returns a naive datetime; subtracting from now raises exactly the error we see: can't subtract offset-naive and offset-aware datetimes.

Fix (one line):

def _parse_iso_ts(raw):
    if not isinstance(raw, str): return None
    try:
        if raw.endswith("Z"): raw = raw[:-1] + "+00:00"
        dt = datetime.fromisoformat(raw)
        if dt.tzinfo is None:
            dt = dt.replace(tzinfo=timezone.utc)   # ← assume UTC for naive ts
        return dt
    except ValueError:
        return None

The query expands ALL forward SOURCES, INVOKES, EXECUTES, CALLS, USES, DEFINES edges out to max_depth=5 by default. On the JGFS_ATMOS_GEMPAK root that quickly explodes into thousands of paths via shared utilities like product_functions.sh, jjob_header.sh, etc. Neptune enforces a 30 s statement timeout and (at depth 5) an OOM ceiling.

The tool already has a graceful-fallback path at depth 2 — that path should become the default behaviour at every depth, with a budgeted traversal.

This is by design for the AgentCore deployment. Runtime containers run in ephemeral Firecracker microVMs and do not bind-mount EFS. Ingestion runs in a separate offline pipeline (scripts/overnight_v17_ingest.sh, overnight_v17_ingest_phase2.sh) using MCP_WORKTREE_ROOT_OVERRIDE=/mnt/efs-staging/supported_repos/global-workflow.

Operational impact:

• Tools that try to read script content (get_job_details(include_content=true), describe_component(show_content=true)) must always fail at runtime.
• Tools that walk git history (_git_head_time, _git_file_time used by check_knowledge_integrity) fall back to staleness-by-days because there is no repo to inspect.

The decoupling is correct architecturally, but tools that advertise content/git capabilities should detect the mount absence and either degrade explicitly or hide the parameter on this deployment.

Five collections (*-nova1024, total 0 docs) are provisioned but unused. Either the Nova embedding profile was a planned alternative that didn't ship, or the ingester was never pointed at it. Worth deciding: keep and populate, or drop and reduce surface area.

get_sdd_framework_status reports 0 workflows; list_sdd_workflows says "/app/sdd_framework/workflows not bind-mounted on hosted Python runtime." The container image was not packaged with the workflow tree, and (like /mnt/workflow) it isn't mounted. SDD session tools work; SDD discovery tools don't.

The prior tool-usage analysis flagged several SoC issues; this health pass adds three more:

1. Health vs readiness are conflated. mcp_health_check reports "HEALTHY" while three production tenants cannot serve a vector search. Health (the service is up) and readiness (this tenant can answer this query) are different concerns; the API conflates them.
2. Ingestion observability is weak. Drift flags exist on the report but aren't surfaced to either MCP consumers (no tool exposes recent ingestion reports) or to operators (no alert fires when documents_created=0). Adding a get_ingestion_status(tenant_id) tool that reads the latest report would close the loop without changing the ingestion architecture.
3. Backend errors leak across the abstraction. Raw OpenSearch NotFoundError strings reach the agent. The error-translation concern should live in one place (the OpenSearch adapter), normalizing all "missing-index for tenant X" errors into a structured, tenant-aware MCP response.

For an operator to reproduce / monitor:

# Health snapshot (run daily; seeds get_health_trend)
mcp_health_check(deep=True)

# Tenant-by-tenant readiness probe
for t in ["gw","gw_sfs","gw_jedi_gfs","gw_v17","gw_gefs_v12"]:
    get_knowledge_base_status(tenant_id=t)
    search_documentation("GEMPAK", tenant_id=t, max_results=1)  # cheap query

# Ingestion report (read directly from EFS until exposed as a tool)
ls -lat /mnt/mdc-mcp-rag/eib-mcp-rag-server/mcp_server_python/scripts/ingestion_reports/

# Integrity check (currently broken — re-run after fix in §3.3)
check_knowledge_integrity(tenant_id="gw")

• MCP utility / GraphRAG tool responses (above).
• Server source: mcp_server_python/src/{tools,data,config,tenancy}/ at /mnt/mdc-mcp-rag/eib-mcp-rag-server.
• Tenant catalog: mcp_server_python/src/config/tenants.yaml.
• Ingestion reports: mcp_server_python/scripts/ingestion_reports/gw_v17_*.json.
• Phase-2 ingest scripts: scripts/overnight_v17_ingest{,_phase2}.sh.