Gemini Embedding Provider Evaluation and Key Request - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

Gemini Embedding 2 Multimodal Provider Evaluation Plan & NOAA API Key Request

Date: July 7, 2026 Author: OMD / EIB MCP-RAG team with assistance from Claude Opus 4.8 xHigh Status: Proposal — pending Gemini API key provisioning Scope: mcp_server_node/ (ingestion) and mcp_server_python/ (runtime) embedding layer

Purpose

Evaluate and integrate Google's gemini-embedding-2 — Google's first natively multimodal embedding model — as a new embedding provider for the MDC MCP-RAG knowledge base, measured head-to-head against the two providers we run today:

titan1024 — Amazon Bedrock Titan Embed Text V2 (1024-dim, the AWS-native default).
mpnet768 — local sentence-transformers/all-mpnet-base-v2 (768-dim, the COTS baseline).

gemini-embedding-2 maps text, images (PNG/JPEG), audio, video, and PDF into one unified vector space, which unlocks a capability neither incumbent has: embedding *.png (and other media) from the ingestion CLI and retrieving it with a text query (cross-modal search). It also brings an 8,192-token context window and Matryoshka output dimensions (128–3072, default 3072).

Scope decision (2026-07-07): this evaluation is centered exclusively on gemini-embedding-2. The earlier, text-only gemini-embedding-001 is not pursued — gemini-embedding-2 supersedes it (multimodal, longer context, server-side normalization), so carrying both added complexity for no benefit.

Why gemini-embedding-2 (verified facts)

Source: Google Gemini API embeddings docs (ai.google.dev/gemini-api/docs/embeddings), verified 2026-07-07.

Property	Value
Model ID	`gemini-embedding-2` (GA) / `gemini-embedding-2-preview`
Endpoint	`POST …/v1beta/models/gemini-embedding-2:embedContent`
Modalities	text, image (PNG/JPEG, ≤6/req), audio (MP3/WAV ≤180s), video (MP4/MOV ≤120s), PDF (≤6 pages) → one unified space
Max input	8,192 tokens
Output dims	128–3072; default 3072; recommended 768/1536/3072 (Matryoshka)
Normalization	auto-normalized at every dimension (no client-side L2 needed)
Task control	`task:` / `title:` text-prefix instructions (asymmetric retrieval)
Auth	API key in the `x-goog-api-key` header

Background — how embeddings are generated today

The embedding layer is a small provider abstraction shared by both servers, with two independent switches: --backend (aws=OpenSearch+Neptune / cots=ChromaDB+Neo4j) selects where vectors are stored; --model selects what generates the vector. Ingestion funnels through one backend-agnostic call site, mcp_server_node/scripts/ingestion_base.py::run() → self.provider.embed([chunk.text]).

Concern	File(s)
Model registry (`ModelProfile`)	`*/embedding_registry.py` (Node scripts + Python src/data)
Provider ABC + `create_provider` factory	`*/embedding_provider.py`
Text ingestion call site	`ingestion_base.py::run()`
Node runtime (query-time) path	`mcp_server_node/src/utils/embeddings.js`

Adding gemini-embedding-2 is additive: two registry profiles, one factory dispatch arm, and a GeminiProvider class with text and image paths. The text ingestion loop is unchanged; image ingestion is a new, additive media route.

Proposed change

Registry profiles (both copies)

ModelProfile(short_name="gemini2_3072", provider="gemini",
             model_id="gemini-embedding-2", dimensions=3072,
             supports_multimodal=True, supports_matryoshka=True,
             provider_params={"output_dimensionality": 3072})
ModelProfile(short_name="gemini2_768", provider="gemini",
             model_id="gemini-embedding-2", dimensions=768,
             supports_multimodal=True, supports_matryoshka=True,
             provider_params={"output_dimensionality": 768})

The server default stays titan1024; the gemini2_* profiles are opt-in via --model.

`create_provider` dispatch (both copies)

if profile.provider == "gemini":
    return GeminiProvider(profile)

`GeminiProvider` — text + image, stdlib `urllib` (zero new dependencies)

class GeminiProvider(EmbeddingProvider):
    """gemini-embedding-2 multimodal embeddings via the Generative Language
    REST API (stdlib urllib). Text + image into one unified vector space.
    The model auto-normalizes every dimension, so no client-side L2 is done."""

    _MAX_RETRIES = 3
    _BACKOFF_S = (1.0, 2.0, 4.0)
    _RETRYABLE_STATUS = frozenset({429, 500, 502, 503, 504})
    _IMAGE_MIME = frozenset({"image/png", "image/jpeg"})
    _ENDPOINT = (
        "https://generativelanguage.googleapis.com/v1beta/models/{model}:embedContent"
    )

    def __init__(self, profile):
        key = os.getenv("GEMINI_API_KEY") or os.getenv("GOOGLE_API_KEY")
        if not key:
            raise EmbeddingError("GEMINI_API_KEY (or GOOGLE_API_KEY) is not set")
        self._profile = profile
        self._key = key
        self._url = self._ENDPOINT.format(model=profile.model_id)
        pp = profile.provider_params
        self._out_dim = int(pp.get("output_dimensionality", profile.dimensions))
        # Asymmetric retrieval instructions (Gemini Embedding 2 uses text prefixes)
        self._doc_instruction = pp.get("doc_instruction", "title: none | text: {text}")
        self._query_instruction = pp.get(
            "query_instruction", "task: search result | query: {text}")

    def embed(self, texts, is_query=False):
        instr = self._query_instruction if is_query else self._doc_instruction
        return [self._embed_part({"text": instr.format(text=t)}) for t in texts]

    def embed_image(self, image_bytes, mime_type="image/png"):
        if mime_type not in self._IMAGE_MIME:
            raise EmbeddingError(
                f"unsupported image mime_type '{mime_type}'; use image/png or image/jpeg")
        b64 = base64.b64encode(image_bytes).decode("ascii")
        return self._embed_part(
            {"inline_data": {"mime_type": mime_type, "data": b64}})

    @property
    def dimensions(self):
        return self._profile.dimensions

    def _embed_part(self, part):
        body = {"content": {"parts": [part]},
                "output_dimensionality": self._out_dim}
        data = self._post(body)
        vector = list(data["embedding"]["values"])  # already unit-normalized
        if len(vector) != self._profile.dimensions:
            raise EmbeddingError(
                f"gemini-embedding-2 returned {len(vector)} dims, "
                f"expected {self._profile.dimensions}")
        return vector

    def _post(self, body):
        import urllib.error
        import urllib.request
        payload = json.dumps(body).encode("utf-8")
        last_exc = None
        for attempt in range(self._MAX_RETRIES + 1):  # 4 attempts
            try:
                req = urllib.request.Request(
                    self._url, data=payload, method="POST",
                    headers={"Content-Type": "application/json",
                             "x-goog-api-key": self._key})
                with urllib.request.urlopen(req, timeout=30) as resp:
                    return json.loads(resp.read())
            except urllib.error.HTTPError as exc:
                last_exc = exc
                if exc.code not in self._RETRYABLE_STATUS or attempt >= self._MAX_RETRIES:
                    raise EmbeddingError(
                        f"gemini-embedding-2 embed failed status={exc.code}: "
                        f"{exc.read().decode('utf-8', 'replace')[:200]}") from exc
                time.sleep(self._BACKOFF_S[attempt])
            except Exception as exc:  # URLError, JSON, KeyError
                last_exc = exc
                if attempt >= self._MAX_RETRIES:
                    raise EmbeddingError(
                        f"gemini-embedding-2 embed failed: {exc}") from exc
                time.sleep(self._BACKOFF_S[attempt])
        raise EmbeddingError(f"gemini-embedding-2 embed failed: {last_exc}")

The Node copy adds the same class; its EmbeddingProvider ABC declares embed_image abstract, which this now genuinely implements (no more NotImplementedError stub — we are multimodal).

Image ingestion route (`*.png` on the command line)

The text loop (ingestion_base.py::run()) is unchanged. Image ingestion is a new additive route: an --images <path-or-glob> flag on a multimodal-capable ingester reads each file's bytes, calls provider.embed_image(bytes, mime_type), and upsert_documents the vector with the file path as the source id and a modality="image" metadata field into the same gemini2_* collection — so a text query can retrieve image documents.

What we are requesting (for the NOAA/Google contact)

Which surface — a Gemini Developer API key (Google AI Studio; generativelanguage.googleapis.com) vs Vertex AI (GCP project + service account, no API key). Planning for the Developer-API key path; please confirm which is approved (Vertex is often the gov-preferred surface — see Compliance).
Model access — the key/project must have the Generative Language API enabled with access to gemini-embedding-2 (multimodal).
Billing-enabled (paid) tier, not free — a full-corpus pass will exceed free-tier RPM/daily caps. Please provide the project's RPM / TPM / RPD so we can size ingest concurrency / --delay.
Network egress — confirm the ingest host (AWS EC2 dev box and/or the Parallel Works VM) may reach generativelanguage.googleapis.com:443 (outbound allowlist entry may be needed).
Data-handling confirmation — embedding sends document text and image bytes to Google's API. Our corpus is public NOAA Global Workflow code/docs, but please confirm outbound-to-Google is permitted; note that on the paid Developer tier (and always on Vertex AI) Google does not use the data to train models, whereas the free tier may.

Compliance & governance

Corpus sensitivity: public global-workflow source/docs — no CUI/PII. Route the request through the normal data-handling sign-off regardless.
Training-data use: request the paid Developer tier or Vertex AI for the "not used for training" guarantee.
FedRAMP context: this is an evaluation, not a production authorization. Any production adoption needs the same FedRAMP-boundary scrutiny documented for the AgentCore path (see MCP-External-Access-FedRAMP-Readiness-Review).
Key storage: GEMINI_API_KEY lives in the shell env / AWS Secrets Manager only — never committed.

Integration gotchas

No client-side normalization. gemini-embedding-2 auto-normalizes every dimension (including truncated 768/1536), so the provider returns vectors as-is — do not re-normalize (that was only needed for the older 001 model).
Task control is a text prefix. Documents embed as title: {t} | text: {c}; queries as task: search result | query: {q}. This asymmetric formatting matters for retrieval quality; the provider supports both via embed(..., is_query=).
Image limits. ≤6 images/request; PNG/JPEG; base64 inline_data part.
Context. 8,192-token input cap — comfortably covers current chunk sizes.
Unified collection. Text and image vectors share the gemini2_* collection so cross-modal (text→image) retrieval works.
Dimensions. gemini2_3072 (flagship) vs gemini2_768 (Matryoshka, cheaper storage) — each model gets its own collection; compare by retrieval metrics.

Running the comparison

export GEMINI_API_KEY=…          # provisioned key
# text corpus
python3.12 mcp_server_node/scripts/ingest_documentation_v8.py \
    --model gemini2_3072 --backend aws --delay <sized-to-RPM>
# images (new multimodal route)
python3.12 mcp_server_node/scripts/ingest_documentation_v8.py \
    --model gemini2_3072 --backend aws --images "docs/diagrams/*.png"

Then:

mcp_server_node/scripts/benchmark_runner.py — P@5 / MRR for gemini2_3072 vs the titan1024 baseline (and mpnet768) on the same query set.
Cross-modal smoke — issue a text query and confirm an ingested image is retrievable.

MCP-Health-Status-Comparative-Report — current dual-provider (Titan / MPNet) status.
MCP-External-Access-FedRAMP-Readiness-Review — federal-authorization context for production adoption.

Implementation is tracked as Kiro spec gemini-embedding-provider with a sister SDD workflow (Phase 66) in the MCP-RAG server repo.