agentcore-mcp-rag — Tool Usage Analysis (v17 GEMPAK task)

Task: Determine the purpose and finer details of ecf/scripts/gfs/product/atmos/gempak/gempak/jgfs_atmos_gempak_master.ecf on the GFS v17 branch, using the agentcore-mcp-rag MCP to its fullest extent. Tenant under test: gw_v17 (dev/gfs.v17) — a multi-tenant exercise. Date: 2026-06-10

This page is the running record of every MCP tool call made during the investigation, how useful each was, and a separation-of-concerns (SoC) assessment with improvement suggestions. The content findings live in the companion page jgfs_atmos_gempak_master.ecf (v17) Analysis.

---

1. Running record of tool calls

Legend — Usefulness: [HIGH] direct answer / [MED] partial or supporting / [LOW] little value / [FAIL] errored or unavailable.

#	Tool	Args (tenant gw_v17 unless noted)	Result	Usefulness
1	get_workflow_structure	—	Tool not available in session	[FAIL]
2	get_knowledge_base_status	—	Tool not available in session	[FAIL]
3	analyze_code_structure	ecf path	Not available (1st); later "File not found" for dev/jobs/...	[FAIL]
4	find_dependencies	ecf path	Not available on ecf path	[FAIL]
5	explain_with_context	topic	Tool not available in session	[FAIL]
6	get_server_info	—	Tool not available	[FAIL]
7	mcp_health_check	—	Tool not available	[FAIL]
8	get_code_context	jgfs_atmos_gempak_master	Typed GW_V17_File; 5 config semantic snippets (config.gempak/atmos_products/wavegempak)	[MED]
9	search_architecture	query	ERROR — index gw_v17_community-summaries not found	[FAIL]
10	explain_workflow_component	jgfs_atmos_gempak_master.ecf	No doc/graph hits	[LOW]
11	get_change_impact	JGFS_ATMOS_GEMPAK (behavior)	LOW 0.10, 0 dependents (leaf)	[MED]
12	find_similar_code	query	ERROR — index gw_v17_code-with-context not found	[FAIL]
13	trace_data_flow	JGFS_ATMOS_GEMPAK	11 outgoing rels (INVOKES/SOURCES nawips, jjob_*, set_strict)	[HIGH]
14	find_related_files	dev/jobs/JGFS_ATMOS_GEMPAK	0 related files	[LOW]
15	find_env_dependencies	FHR_LIST	0 direct dep/export; GGSR neighbours (prod/atmanlprod tasks)	[MED]
16	trace_full_execution_chain	JGFS_ATMOS_GEMPAK (depth 5, then 2)	ERROR — Neptune 500 out of memory (both)	[FAIL]
17	find_callers_callees	exgfs_atmos_nawips.sh	Caller=J-Job; callee=product_functions.sh; env deps NAGRIB/GEMGRD/WGRIB2/…	[HIGH]
18	find_dependencies	dev/jobs/JGFS_ATMOS_GEMPAK	10 upstream imports, 0 downstream, no cycles	[HIGH]
19	find_env_dependencies	WGF	0 direct; GGSR neighbour exglobal_fsm.sh	[LOW]
20	search_documentation	query	"No results" (no v17 docs index)	[FAIL]
21	get_job_details	JGFS_ATMOS_GEMPAK (content=true)	Metadata + sourced scripts + env vars; content unavailable on hosted port	[MED]
22	analyze_code_structure	dev/jobs/JGFS_ATMOS_GEMPAK	"File not found" (node keyed differently)	[LOW]
23	get_operational_guidance	operation	ERROR — index gw_v17_global-workflow-docs not found	[FAIL]
24	trace_execution_path	exgfs_atmos_nawips.sh (depth 3)	3 callees + GGSR env weighting	[MED]
25	get_code_context	config.gempak	Not found, but useful did-you-mean (C48_ATM/config.gempak, …)	[MED]
26	search_documentation	query, tenant gw	5 hits (UPP, METplus, GW testing docs) — baseline works	[MED]
27	get_code_context	C48_ATM/config.gempak	GW_V17_EXPDIRConfig, GGSR neighborhood + snippets (sim 1.000)	[HIGH]
28	get_change_impact	exgfs_atmos_nawips.sh (signature)	LOW 0.30, 1 dependent (J-Job)	[HIGH]
29	find_env_dependencies	COMOUT_ATMOS_GEMPAK_0p25	GGSR neighbours: JGFS/JGDAS GEMPAK, PGRB2_SPEC, goes_nawips	[HIGH]
30	find_callers_callees	product_functions.sh	5 callers (nawips/goes/awips/gdas/interp), leaf	[HIGH]

Tally

• 30 calls (incl. retries and 1 gw baseline control).
• [HIGH] 7 · [MED] 7 · [LOW] 4 · [FAIL] 12.
• All 7 [HIGH] results came from graph-traversal / GraphRAG tools.
• All 12 [FAIL] results were either vector/OpenSearch tools on gw_v17 (missing indices — Gap B) or tools not registered in this session, or Neptune OOM.

---

2. What worked, what didn't, and why

Worked well (Neptune graph + GraphRAG)

trace_data_flow, find_dependencies, find_callers_callees, trace_execution_path, get_change_impact, and get_code_context (for graph-resident symbols) returned accurate, verifiable relationships for gw_v17. The full execution chain J-Job → exgfs_atmos_nawips.sh → product_functions.sh, the env-var fan-out, and the blast-radius scores were all corroborated against the on-disk source tree. The GGSR weighted-context tables were a genuine bonus — they surfaced sibling consumers (JGDAS_ATMOS_GEMPAK, JGFS_ATMOS_GEMPAK_PGRB2_SPEC) that a plain grep would not rank.

Did not work (and the root cause)

1. Vector/semantic tools on gw_v17 (Gap B). search_documentation, search_architecture, find_similar_code, get_operational_guidance all failed because the prefixed OpenSearch indices (gw_v17_documentation, gw_v17_code-with-context-v8-0-0, gw_v17_community-summaries, gw_v17_global-workflow-docs-v8-0-0) do not exist yet. This is the documented multi-tenant limitation: node ingestion for non-gw tenants is done, but vector/relationship porting is still in progress. The gw baseline control (#26) returned full results, confirming the tools themselves are healthy.
2. trace_full_execution_chain → Neptune OOM. Failed at depth 5 and depth 2. The cross-language chain query is too broad/unbounded for the v17 graph shape and the server's Neptune memory budget. This is the single most disappointing failure — it is exactly the tool the task ("trace the whole chain") most wanted.
3. Several utility/info tools "not available" in this session (get_server_info, mcp_health_check, get_workflow_structure, get_knowledge_base_status, explain_with_context, the ecf-path find_dependencies). These were not exposed/registered to the client, so health and high-level structure could not be queried to confirm the index gap directly.

Partial / fragile

• get_job_details returns rich metadata but not script content on the hosted Python port — you must read the file from the repo. The content section is a dead end.
• get_code_context is inconsistent across collections for the same tenant: it returned expdir-config snippets (similarity 1.000) for C48_ATM/config.gempak, yet find_similar_code reported the gw_v17_code-with-context index as missing. Two tools that ostensibly read "v17 vector data" disagree about whether that data exists.
• Path keying is brittle: find_dependencies works on dev/jobs/JGFS_ATMOS_GEMPAK but analyze_code_structure says "File not found" for the same string, and the original ecf/.../*.ecf path is unknown to every graph tool (only the symbol JGFS_ATMOS_GEMPAK resolves).

---

3. Separation-of-Concerns (SoC) assessment

The toolset is organized into clean modules (Workflow Info, Code Analysis, Semantic Search, EE2, Operational, GraphRAG, SDD, GitHub, Utility). The concern boundaries are mostly sound, but this exercise exposed several leaks and overlaps:

3.1 Storage-backend concern leaks into the caller

A consumer should not need to know that "documentation = OpenSearch" and "relationships = Neptune." Today the failure mode is a raw backend error (NotFoundError(404, index_not_found_exception, gw_v17_community-summaries)) bubbled straight to the agent. The data-availability concern is not encapsulated. Suggestion: normalize to a tenant-capability response, e.g. [INFO] Tenant gw_v17 has graph data but no vector indices yet (Gap B). Falling back to graph-only. Add a tenant_capabilities(tenant_id) utility so a consumer can branch before calling — instead of discovering the gap one 404 at a time.

3.2 Overlapping tools with divergent backends

get_code_context, find_similar_code, and search_documentation all answer "what is related to X semantically," but hit different indices and disagree per tenant (§2). This is a single concern (semantic neighborhood) split across three tools with inconsistent contracts. Suggestion: have them share one resolver/availability layer so that, for a given tenant, they agree on what exists and degrade identically.

3.3 Traversal depth/cost is the caller's problem

trace_full_execution_chain OOM-ing at depth 2 means resource governance is not a tool concern today — it is pushed onto the user as trial-and-error. Suggestion: enforce an internal node/edge budget with early truncation and a partial result ([WARN] chain truncated at 500 nodes) rather than a 500 error. Cross-language chain assembly (Shell→Fortran→Python) should be paginated or breadth-capped server-side.

3.4 Identity/addressing concern is inconsistent

Tools variously key on symbol name (JGFS_ATMOS_GEMPAK), repo-relative path (dev/jobs/JGFS_ATMOS_GEMPAK), or bare filename, and acceptance differs per tool (§2). Resolution (name → node) is a cross-cutting concern that should live in one place. Suggestion: a shared resolver accepting any of {symbol, path, filename} with fuzzy fallback (the did-you-mean in get_code_context is the right pattern — generalize it to all graph tools).

3.5 Content vs metadata concern

get_job_details(include_content=true) advertises content it cannot deliver on the hosted port. Suggestion: either drop the parameter on that deployment or have it transparently read from the mounted worktree, so the "give me the script" concern is satisfied by one tool rather than forcing a separate read.

3.6 Tool registration / discoverability

A third of the catalog was "not available" in-session with no way to enumerate what was available (because get_server_info/mcp_health_check were themselves unavailable). Suggestion: always expose the Utility module (server info + health) so a consumer can discover capabilities and tenant data-readiness up front — discoverability is its own concern and should never be among the unavailable tools.

---

4. Updated usefulness verdict

Module	Verdict for this task	Notes
GraphRAG (trace_data_flow, get_change_impact, get_code_context)	Strong	Carried the analysis; accurate, verifiable, value-add GGSR context.
Code Analysis (find_dependencies, find_callers_callees, trace_execution_path)	Strong	Reconstructed the full call/env graph for v17.
Semantic Search (vector)	Unusable on gw_v17	Missing indices (Gap B); works on gw.
Operational	Unusable on gw_v17	Same missing-index cause.
trace_full_execution_chain	Failed	Neptune OOM at any depth — needs budgeting.
Utility / Info	Absent	Not registered; blocked health/capability checks.

Bottom line for multi-tenant work: for any tenant other than gw, treat agentcore-mcp-rag as a graph-only service today. Lead with GraphRAG and Code-Analysis tools, keep symbol names as the addressing key, cross-check against the on-disk worktree, and expect every vector/operational/architecture tool to 404 until Gap B index porting lands for gw_v17. The graph tools alone were sufficient to fully and correctly characterize jgfs_atmos_gempak_master.ecf.

---

5. Concrete improvement backlog (prioritized)

1. Encapsulate tenant data-readiness — tenant_capabilities() + graceful graph-only fallback instead of raw 404s. (fixes 3.1, biggest UX win)
2. Budget trace_full_execution_chain — node/edge cap, partial results, no OOM. (3.3)
3. Unify symbol/path resolution across graph tools with fuzzy did-you-mean. (3.4)
4. Always expose Utility module (server info, health) for discoverability. (3.6)
5. Reconcile semantic tools so get_code_context/find_similar_code/search_documentation agree per tenant. (3.2)
6. Fix get_job_details content on the hosted port or remove the misleading flag. (3.5)

Content was rephrased for compliance with licensing restrictions where source material was quoted.