Version: 0.2.0 (Ground-Truth Refresh) Status: Initiative — pilot target identified Created: 2026-05-06 Updated: 2026-05-22 (folded in five-pillar reality discovered during dev/sfs deep dive) Author: Terry McGuinness + AI Assistant Type: Initiative (multi-phase; spawns phase54a_* … phase54f_* execution specs) Dependency: Phase 11E (n8n / orchestration), Phase 26 (Docker MCP Gateway), Phase 53 (Gateway Tool Quality) Related: Phase 4d (Multi-tenant SDD workspaces), Phase 52 (v17 paradigm coverage)

The v0.1 draft (2026-05-06) described a 4-tenant target — gw-mcp / gw-v17-mcp / ufs-mcp / mpas-mcp — based on partial visibility into the OMD program portfolio. The 2026-05-22 deep dive on the freshly connected global-workflow_dev-sfs submodule surfaced ground truth that materially expands the picture: five active development pillars exist inside NOAA-EMC/global-workflow.git alone, and develop is no longer the canonical view of "global-workflow" — it is one of six living trajectories. This refresh folds that reality into the Initiative and identifies dev/sfs as the natural pilot target.

The NOAA Office of Modeling Development (OMD, formerly EMC) maintains a portfolio of modeling programs that increasingly need first-class AI/RAG support — not just the global-workflow:develop line that the current agentcore-mcp-rag server was built for. Discovered during the 2026-05-22 dev/sfs investigation, the live portfolio inside the single NOAA-EMC/global-workflow.git repository is:

Beyond global-workflow, the OMD portfolio also includes UFS (a shared component embedded inside multiple downstream programs), MPAS (which contains UFS components), and future programs (RRFS, HAFS, …) as the portfolio grows.

The current architecture has one monolithic AgentCore Runtime (<PYTHON_RUNTIME_ID>) ingesting one branch of one repo into one Neptune cluster and one set of OpenSearch indices. There is no way to ask the platform "reason about the gfs.v17 coupled branch" or "reason about MPAS" without re-pointing the entire server. The configuration-management hazard the v0.1 draft warned about is already real: agents asking about JGLOBAL_FORECAST get develop-branch answers when the user is working from dev/gfs.v17 where the same file may not exist or has materially different behaviour.

This Initiative establishes the architecture and execution plan to decompose that monolith into program-scoped MCP tenants — one per OMD pillar, with shared-component inheritance, branch-scoped isolation, content-addressed storage dedupe, lifecycle states, and a router that helps agents pick the right tenant.

A secondary capability (third-party MCP discovery with a headless secret broker) is included because it shares the catalog/auth machinery that the multi-tenant arm needs and because it unlocks several adjacent OMD workflows (literature review, external dataset ingestion). It is intentionally framed as supporting, not leading.

As a repo configuration manager juggling six living branches across global-workflow (develop, dev/sfs, dev/gfs.v17, dev/gcafs.v1, dev/gfs.v16, dev/jedi-gfs) plus ufs and future programs, I want one gateway URL where each pillar is a distinct MCP tenant (gw-mcp, gw-sfs-mcp, gw-v17-mcp, gw-gcafs-mcp, gw-v16-mcp, gw-jedi-mcp, ufs-mcp, mpas-mcp) with its own data view, and where shared components (UFS) are referenced from a single source of truth, so that an agent working a v17-coupled bug never sees develop-branch code, an SFS user gets the post-processing pipeline that doesn't yet exist on develop, an MPAS agent transparently inherits UFS tools, and divergence between branches stops causing cross-contaminated answers.

As an OMD program lead onboarding a new model (e.g., MPAS, RRFS, HAFS) or a new branch within an existing program (e.g., the next dev/* line), I want a documented "tenant template" that defines what a new pillar needs (repo reference, branch, optional extends: parent, lifecycle state, staleness threshold, auth requirements), so that onboarding a new pillar is a configuration change, not a code change to the MCP server.

As an OMD researcher writing a forward-looking technical document, I want the agent to autonomously discover, authenticate to, and call third-party knowledge MCPs (arXiv, Semantic Scholar, OpenAlex) through the same gateway, so that literature review and citation discovery are part of the same conversation as the code analysis, without me hand-managing API keys.

• Cross-tenant query federation — "find this symbol across UFS and MPAS".
• Tenant lifecycle CLI — eib-tenant create gw-sfs --extends gw --branch dev/sfs --lifecycle experimental.
• Audit log — who-invoked-which-tool-on-which-tenant.
• Per-MCP rate-limit / cost caps at the broker layer (54f).
• OAuth 2.0 device-code flow for third-party MCPs needing interactive consent.
• Cross-pillar diff tooling — "show me what dev/gfs.v17 changed in the forecast driver" implemented as a tenant-pair query.

• Re-introducing Docker Desktop on the gateway host.
• Re-implementing the existing 52-tool surface — 54b re-packages the existing tools to be tenant-aware; their behaviour is unchanged.
• Migrating away from Neptune or OpenSearch — 54b uses native index-prefix and label-prefix scoping.
• Onboarding non-OMD programs in v1.
• Any third-party MCP requiring a paid plan beyond a free tier.

1. The catalog lists ≥6 OMD pillar tenants by end-of-Initiative: gw-mcp (develop), gw-sfs-mcp, gw-v17-mcp, gw-gcafs-mcp, gw-v16-mcp, plus ufs-mcp. (gw-jedi-mcp and mpas-mcp may be staged per R5.)
2. Branch-isolation probe: querying analyze_code_structure on dev/scripts/exglobal_atmos_post.sh returns the SFS post-processing implementation from gw-sfs-mcp and a "file not found" from gw-mcp (the file does not exist on develop).
3. Inheritance probe: mpas-mcp.find_callers_callees on a UFS symbol returns results without ufs-mcp being explicitly invoked.
4. Tenant attribution: every response from any tenant carries a tenant_id field; audit-log entries reference it.
5. CM-hazard regression: an agent prompt "summarize the SFS atmos post pipeline" returns SFS-pillar code only; never returns content from develop (verified by file-path inspection — files under dev/jobs/JGLOBAL_ATMOS_POST only exist on dev/sfs).
6. Onboarding cost: adding a seventh tenant (e.g., RRFS) is documented as a catalog-only change; no edits to the MCP server source code.
7. (NEW) Staleness flag: querying gw-jedi-mcp (5 mo stale) returns a [STALE] marker on every response with the last-commit timestamp.
8. (NEW) Promotion: when a dev/* branch merges to develop, the corresponding tenant transitions to lifecycle: merged; subsequent calls return a deprecation notice pointing at gw-mcp.

9. mcp-find arxiv on the production gateway returns ≥1 result.
10. mcp-config-set rejects a config that omits a required secret with a schema-validation error.
11. The agent runs ≥10 chained search_papers / get_paper_details calls against semantic-scholar-mcp without manual key intervention.
12. Gateway restart preserves the catalog, the secret store, and any saved tenant configurations. No manual re-bootstrap.

   54a (catalog schema) ─┐
                         ├─► 54b (data isolation + tenant_id propagation)
                         │            │
                         │            ├─► 54c (extends:)
                         │            └─► 54d (branch tenants + dedupe) ──► 54g (lifecycle + staleness)
                         │                       │
                         │                       ▼
                         └────────────────────► 54e (routing & attribution)
                         │
                         └─► 54f (auth broker + 3rd-party pilot)   [parallel]

Pilot rollout sequence (refined for incremental validation):

1. omd-tenants-1-foundation = 54a + 54b minimum. Existing AgentCore Runtime becomes the single tenant gw; nothing else changes from the user perspective but the plumbing is in place.
2. omd-tenants-2-sfs-pilot = 54d against the smallest pillar. Stand up gw-sfs against supported_repos/global-workflow_dev-sfs, ingest only the 112-file diff, run the branch-isolation probe.
3. gw-v17-tenant = the operational prize. Repeat for dev/gfs.v17 once the SFS pilot proves the foundation.
4. Remaining pillars = gw-gcafs-mcp, gw-v16-mcp, gw-jedi-mcp in order of community urgency.
5. extends: (54c) + UFS tenant = once 2 + pillars share UFS, dedupe is worth the engineering.
6. Routing + lifecycle (54e + 54g) = once 3 + tenants exist, the UX work pays off.
7. Auth broker (54f) = parallel to all of the above.

Each will get its own SDD spec under sdd_framework/workflows/:

phase54a_tenant_catalog_schema.md
phase54b_per_tenant_data_isolation.md
phase54c_shared_component_inheritance.md
phase54d_branch_scoped_tenants.md
phase54e_tenant_routing_and_attribution.md
phase54f_headless_auth_broker_and_3p_mcp_pilot.md
phase54g_tenant_lifecycle_and_staleness.md

Use start_sdd_session({phase:"phase54_omd_multi_pillar_mcp_initiative", role:"initiative"}) to open the umbrella session. Each sub-phase opens its own session and links back to this Initiative via the parent_phase field.

• .kiro/settings/mcp.json — current single-tenant gateway config
• supported_repos/global-workflow — develop (current single tenant)
• supported_repos/global-workflow_dev-sfs — first multi-tenant pilot target
• phase4d_multi_tenant_sdd_workspaces.md — prior multi-tenant work (SDD-workspace scope)
• phase52_v17_paradigm_coverage.md — earlier work specifically on v17 paradigm gaps
• v17_paradigm_assessment.md — 2026-04-22 v17 branch-comparison

# .kiro/settings/agentcore_tenants.yaml (illustrative, post-54a/54b/54d)

defaults:
  runtime_image: <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/mdc-mcp-rag:python-titan-v5
  staleness_threshold_days: 60

tenants:
  # === In-repo pillars (NOAA-EMC/global-workflow.git) ===

  gw:
    tenant_id: gw
    repo_ref: NOAA-EMC/global-workflow
    branch: develop
    index_prefix: gw_
    label_prefix: GW_
    lifecycle: production
    description: "Operational GFS/GEFS line — current canonical baseline"

  gw-sfs:
    tenant_id: gw_sfs
    repo_ref: NOAA-EMC/global-workflow
    branch: dev/sfs
    extends: [gw]                # Inherit develop's untouched code
    index_prefix: gw_sfs_
    label_prefix: GW_SFS_
    lifecycle: experimental
    description: "Seasonal Forecast System post-processing pipeline (atmos/ocean/ice + ocean diags)"

  gw-v17:
    tenant_id: gw_v17
    repo_ref: NOAA-EMC/global-workflow
    branch: dev/gfs.v17
    extends: [gw]
    index_prefix: gw_v17_
    label_prefix: GW_V17_
    lifecycle: staging
    description: "Next-generation coupled GFS — operational target"

  gw-gcafs:
    tenant_id: gw_gcafs
    repo_ref: NOAA-EMC/global-workflow
    branch: dev/gcafs.v1
    extends: [gw]
    index_prefix: gw_gcafs_
    label_prefix: GW_GCAFS_
    lifecycle: experimental
    description: "Global Chemistry & Aerosol Forecast System v1"

  gw-v16:
    tenant_id: gw_v16
    repo_ref: NOAA-EMC/global-workflow
    branch: dev/gfs.v16
    extends: [gw]
    index_prefix: gw_v16_
    label_prefix: GW_V16_
    lifecycle: production
    description: "Current operational v16 line (parallel evolution)"

  gw-jedi:
    tenant_id: gw_jedi
    repo_ref: NOAA-EMC/global-workflow
    branch: dev/jedi-gfs
    extends: [gw]
    index_prefix: gw_jedi_
    label_prefix: GW_JEDI_
    lifecycle: experimental
    staleness_marker: true       # 5 mo idle as of 2026-05-22
    description: "JEDI-based DA experimentation (currently stale)"

  # === Cross-program pillars (future) ===

  ufs:
    tenant_id: ufs
    repo_ref: ufs-community/ufs-weather-model
    branch: develop
    index_prefix: ufs_
    label_prefix: UFS_
    lifecycle: production
    description: "Shared atmosphere/ocean/ice/wave coupled component"

  mpas:
    tenant_id: mpas
    repo_ref: MPAS-Dev/MPAS-Model
    branch: main
    extends: [ufs]
    index_prefix: mpas_
    label_prefix: MPAS_
    lifecycle: experimental
    description: "Model for Prediction Across Scales — voronoi-mesh dynamical core"

Key property: all tenants run the same Docker image. The 52-tool surface is identical; only the data view differs. CM divergence between six pillars becomes a catalog concern, not a server-code concern. Onboarding a new OMD program is adding ~10 lines of YAML.

The dev/sfs branch is small enough to ingest in one Bedrock embedding run, in active code review (so the lifecycle promotion drill is imminent), and tightly synced with develop (so dedupe shows its value immediately). It is the canonical "learn the foundation cheaply, then apply it to the operational prize" target.