SageMaker GraphRAG Production Pipeline Strategy - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

SageMaker GraphRAG Production Pipeline Strategy

Date: April 14, 2026 Author: Terrence McGuinness, NOAA MDC Status: Infrastructure Built — Activation Deferred to Production Phase Phase: Post-Phase 52 (Bedrock Re-Ingestion)

Executive Summary

The MDC MCP RAG platform has a fully built but not yet activated Amazon SageMaker Processing pipeline designed to offload compute-intensive embedding generation, model fine-tuning, and automated knowledge graph maintenance to managed AWS infrastructure. This document explains what has been built, why activation is deferred, and when it becomes the right investment.

The SageMaker infrastructure is a strategic asset for the production self-healing GraphRAG loop. It is not needed for the current one-time Bedrock Titan re-ingestion (which runs in 30 minutes on the existing EC2 for under $1), but becomes essential when the system transitions to automated drift detection, scheduled re-ingestion, and domain-adaptive fine-tuning.

What Has Been Built (Phase 49)

All SageMaker components were implemented during Phase 49 (Ingestion Pipeline Restructure) and are committed on the develop_aws branch:

Component File Purpose
Job Launcher scripts/sagemaker_launcher.py Submit ingestion scripts as SageMaker Processing Jobs with --model, --backend, --collections flags
ECR Dockerfile scripts/Dockerfile.sagemaker Container image packaging all 7 ingestion scripts, embedding registry, and AWS backend adapters
Requirements scripts/requirements-sagemaker.txt Python dependencies (boto3, sentence-transformers, opensearch-py, neo4j, fparser)
Drift Detector scripts/drift_detector.py Sample documents, re-embed, compute cosine similarity, detect stale content
Benchmark Runner scripts/benchmark_runner.py Precision@k, Recall@k, MRR, nDCG evaluation across models and search modes
Fine-Tuning Pipeline scripts/fine_tuning_pipeline.py Generate training pairs, submit SageMaker Training Jobs, register fine-tuned models
Hard Negative Miner scripts/hard_negative_miner.py Graph-powered training triples using Neptune community distance as difficulty signal
Feedback Logger src/data/feedback/FeedbackLogger.js Anonymized query-result pair logging to S3 for training data collection

Embedding Model Registry

The pipeline supports multiple embedding models through a centralized registry:

Model Provider Dimensions Status
mpnet768 Local (sentence-transformers) 768 Active default
titan1024 AWS Bedrock (Titan V2) 1024 Next (Phase 52)
nova256-3072 AWS Bedrock (Nova) 256-3072 Registered, Matryoshka-capable

Why SageMaker Is Deferred (Cost-Benefit Analysis)

Current Re-Ingestion: Bedrock Titan (Phase 52)

The immediate task is re-ingesting 85,921 documents with Bedrock Titan 1024-dim embeddings to benchmark against the current MPNet 768-dim baseline.

Factor EC2 Direct SageMaker
Bedrock API cost $0.86 $0.86 (same)
Compute cost $0.00 (EC2 already running) $0.05 (ml.m5.large)
Setup time 0 hours 2-4 hours
Total cost $0.86 $0.91
Estimated runtime ~30 minutes ~30 minutes

The Bedrock embedding is an API call — the heavy compute happens on Bedrock's side. The EC2 instance is just making HTTP requests and writing results to OpenSearch. SageMaker adds no value for this workload pattern.

Setup Work Required Before SageMaker Activation

These items are tracked but not blocking current development:

  1. Build and push Docker image to ECR (Dockerfile.sagemaker)
  2. Provision SAGEMAKER_ROLE_ARN IAM role with Bedrock, OpenSearch, Neptune, S3 permissions
  3. Fix Neptune IAM auth in Python aws_backend.py (SigV4 signing — currently uses Auth("none"))
  4. Add environment variable passthrough in sagemaker_launcher.py (OPENSEARCH_ENDPOINT, etc.)
  5. Add documentation_sources_config.py to Dockerfile (SPOT config not currently packaged)
  6. End-to-end test with smallest collection (ee2-standards, 34 docs)

When SageMaker Becomes Essential

Phase 1: Automated Drift Detection (Post-Phase 52)

Once Bedrock Titan embeddings are validated, drift detection runs on a schedule:

CloudWatch Events (weekly)
  → SageMaker Processing Job (drift_detector.py)
    → Sample 100 docs per collection
    → Re-embed with current model
    → Compare cosine similarity to stored embeddings
    → If drift > threshold (0.95): trigger re-ingestion job
    → Upload drift report to S3

SageMaker is the right tool here because:

  • Runs on a schedule without keeping an EC2 instance alive
  • IAM-scoped permissions (no long-lived credentials)
  • CloudWatch integration for alerting on drift events
  • Cost: ~$0.02 per weekly run (5 minutes on ml.m5.large)

Phase 2: Domain-Adaptive Fine-Tuning

The self-improving loop uses SageMaker Training Jobs:

Feedback Logger (S3 JSON Lines)
  → Hard Negative Miner (Neptune graph distance)
    → Generate (anchor, positive, hard_negative) triples
      → SageMaker Training Job (fine_tuning_pipeline.py)
        → Fine-tune base model on domain-specific pairs
          → Register new model in EmbeddingModelRegistry
            → Re-ingest with fine-tuned model
              → Benchmark against baseline

SageMaker is essential here because:

  • GPU instances (ml.g5.xlarge) for model training — EC2 t3.xlarge has no GPU
  • Managed training lifecycle (checkpointing, spot instances, early stopping)
  • Model artifact storage in S3 with versioning
  • Cost: ~$5-15 per training run (1-2 hours on ml.g5.xlarge)

Phase 3: Production Self-Healing GraphRAG

The full production loop combines all components:

┌─────────────────────────────────────────────────────────┐
│              Production Self-Healing Loop                │
│                                                         │
│  ┌──────────┐    ┌──────────────┐    ┌──────────────┐   │
│  │  Drift   │───→│  SageMaker   │───→│  Benchmark   │   │
│  │ Detector │    │  Re-Ingest   │    │   Runner     │   │
│  └────┬─────┘    └──────────────┘    └──────┬───────┘   │
│       │                                      │          │
│       │          ┌──────────────┐             │          │
│       └──────────│  Feedback    │◄────────────┘          │
│                  │   Logger     │                        │
│                  └──────┬───────┘                        │
│                         │                               │
│                  ┌──────▼───────┐    ┌──────────────┐   │
│                  │ Hard Neg     │───→│  Fine-Tune   │   │
│                  │   Miner      │    │  (SageMaker) │   │
│                  └──────────────┘    └──────────────┘   │
│                                                         │
│  Trigger: CloudWatch Events (weekly/monthly)            │
│  Compute: SageMaker Processing + Training Jobs          │
│  Storage: S3 (reports, models, feedback logs)           │
│  Graph:   Neptune community refresh on model update     │
└─────────────────────────────────────────────────────────┘

This is where the SageMaker investment pays off — no manual intervention, the system detects when its knowledge is stale, re-ingests with the best available model, and continuously improves retrieval quality through domain-specific fine-tuning.

Cost Projections

Scenario Frequency SageMaker Cost Bedrock Cost Total
Drift detection only Weekly $1/month $0 $1/month
Drift + re-ingest (1 collection) Monthly $2/month $0.20 $2.20/month
Full re-ingest (all 5 collections) Quarterly $0.50/run $0.86/run $1.36/quarter
Fine-tuning cycle Quarterly $10-15/run $0 $10-15/quarter
Full self-healing loop Monthly ~$5/month ~$1/month ~$6/month

All costs assume on-demand pricing. Spot instances reduce SageMaker Training costs by 60-70%.

Relationship to Current Development Priorities

Priority Phase SageMaker Role
1. Neptune GGSR compatibility Phase 51c None — code fix only
2. Bedrock Titan re-ingestion Phase 52 Deferred — run on EC2
3. AgentCore MCP deployment Phase 51b None — server deployment
4. Drift detection activation Post-52 First SageMaker use case
5. Fine-tuning pipeline Post-52 GPU training jobs
6. Production self-healing Production Full SageMaker integration

Summary

The SageMaker pipeline infrastructure is built, tested in dry-run, and ready for activation when the use case justifies the setup investment. The current development phase focuses on getting the GraphRAG access patterns working correctly on Neptune (GGSR compatibility) and validating Bedrock Titan embeddings — both of which run efficiently on the existing EC2 instance.

SageMaker becomes the right tool when we move from ad-hoc development to automated production operations: scheduled drift detection, triggered re-ingestion, and continuous model improvement through the self-healing feedback loop.

Bottom line: We have the infrastructure. We activate it when the workload demands it.