How the GGSR Search System Works - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

How the GGSR Search System Works

When a developer asks a question, the system needs to decide: is this a question about code structure (like "what calls this function?") or about documentation (like "how does the forecast model work?"). It handles each differently, then combines the results.

The three pieces

1. Graph Traversal — walks the code relationship graph

Think of the codebase as a map of connections. Function A calls Function B, which uses Module C, which imports Library D. The graph traversal starts at a symbol you care about and walks outward through these connections, collecting nearby neighbors.

It assigns a relevance score to each neighbor based on two things: the type of connection (a direct function call is more relevant than a distant import) and how many hops away it is (closer neighbors score higher). This is the "hop decay" — each step away from the starting point reduces the score.

This piece only looks at structure. It doesn't read any text or understand meaning.

2. Fusion Layer — combines structure with meaning

This is where the graph results meet the text search. It takes the structural neighbors from step 1 and asks: "do any of these have documentation or code snippets that are semantically similar to the question?"

It also decides how to balance the two sources. A question like "what calls setuprad?" leans heavily on the graph. A question like "how does data assimilation work?" leans heavily on text search. The system classifies the query and adjusts the mix.

3. Hybrid Text Search — smart keyword + vector search

For the text search side, the system runs two searches in parallel against the document store:

  • A keyword search that matches exact terms (good for function names, file paths, variable names)
  • A vector search that matches meaning (good for natural language questions)

It merges the results. If the query looks like a code identifier — camelCase, snake_case, or a file path — the keyword search gets a boost. If it looks like a natural language question, the vector search leads.

After the text results come back, the system optionally expands them by looking up their graph neighbors. If a document mentions setuprad, the system can pull in the functions that call it and the modules it depends on, giving richer context.

How they fit together

Developer asks a question
        │
        ▼
  ┌─────────────┐
  │  Classify   │  Is this about code structure or documentation?
  │  the query  │
  └──────┬──────┘
         │
    ┌────┴────┐
    ▼         ▼
 Structure   Meaning
  heavy       heavy
    │         │
    ▼         ▼
  Graph     Text Search
  walk      (keyword + vector)
    │         │
    ▼         ▼
  Nearby    Relevant
  code       docs
    │         │
    └────┬────┘
         ▼
  ┌─────────────┐
  │   Combine   │  Merge, re-rank, return top results
  │   results   │
  └─────────────┘

A concrete example

Say you ask: "What environment variables does the forecast job depend on?"

  1. The system classifies this as a structure-heavy query (it's about dependencies, not documentation).

  2. The graph traversal finds the exglobal_forecast.sh script node, then walks its DEPENDS_ON_ENV edges to find variables like PDY, RUN, CASE, DELTIM, APRUN_UFS, and others.

  3. The text search looks for documents that discuss forecast environment configuration, returning relevant sections from the workflow docs.

  4. The fusion layer combines both: the graph gives you the exact variable list with structural proof, the text search gives you the human-written explanation of what those variables mean.

The result is more complete than either source alone.

Why the embedding model doesn't matter here

The graph traversal doesn't use embeddings at all — it walks connections between code entities. The text search uses whatever embedding model is configured at the index level. Swapping from one model to another just means pointing at a different search index. The graph side and the fusion logic stay the same.

Both the 768-dimension and 1024-dimension indices work with the same pipeline. The graph provides structural context, the text search provides semantic similarity, and the fusion layer combines them regardless of which embedding model produced the vectors.

What's running today

The full pipeline is operational on AWS:

  • Graph database: Amazon Neptune with 148,723 nodes and 2,820,440 relationships across Shell, Python, and Fortran code
  • Text search: Amazon OpenSearch with 206,341 documents across 17 indices (multiple embedding models)
  • Hosting: AWS Bedrock AgentCore Runtime, accessible through 51 MCP tools via Kiro IDE

May 2026 — MDC MCP-RAG Platform