Setting Up Google Content Providers - ericfitz/tmi GitHub Wiki

Setting Up Google Content Providers

This page is an operator walkthrough for configuring TMI's two Google content providers:

  • google_drive — service-account access to Google Drive files explicitly shared with TMI's bot identity.
  • google_workspace — delegated, per-user access to a user's own Drive files via OAuth + the Google Picker.

The two providers solve different problems and use different parts of the Google Cloud Console. The first three steps are shared (Cloud project setup); after that the two diverge. You can run either one alone, or both at the same time — the registry is wired so the per-user google_workspace source wins for picker-attached docs and google_drive covers everything else.

For the architectural background of the delegated model (token storage, refresh, link/unlink endpoints, lifecycle), see the Delegated Content Providers page. This page is the operator setup guide for Google specifically.

Choosing between the two

Provider Auth model Who can read the doc When to use
google_drive (service-account) One bot/service account, no end-user OAuth Only docs explicitly shared with the SA's email Shared corporate drives where ops can grant the SA access; works without users having to link Google.
google_workspace (delegated) Per-user OAuth + Google Picker SDK in the UI The end-user's own Drive access (drive.file scope: only files they pick) Per-user content where each TMI user attaches their own Drive docs. Recommended for production.

You almost never want both pointed at the same documents. Pick one, or run both with disjoint document sets.

Step 1 — Cloud project setup (shared)

Open the Google Cloud Console and either pick an existing project or create a new one. Note the project ID (and the project number — you'll need that for google_workspace).

Enable the Google Drive API for the project: APIs & Services → Library → search Google Drive API → Enable.

If you plan to enable google_workspace, also enable the Google Picker API in the same library.

Step 2 — google_drive (service-account) setup

2a. Create the service account

  1. IAM & Admin → Service Accounts → Create service account.
  2. Give it a name (e.g. tmi-content-gdrive). Note the generated email — it will look like tmi-content-gdrive@<project>.iam.gserviceaccount.com. This is the service_account_email value.
  3. Skip the "Grant access" step — Drive permissions are per-document, not project-IAM.
  4. After creation, click into the SA → Keys → Add Key → Create new key → JSON. A .json file downloads. This is the credentials_file.
  5. Move the JSON file somewhere TMI can read it. Treat it as a secret. Never commit it to source control.

2b. Grant the SA access to your documents

The SA can only read Drive files explicitly shared with its email address.

  • Per-document: in Drive, right-click the file → Share → paste the SA email → set Viewer (TMI only reads).
  • Per-shared-drive: in Drive → Shared drives → pick a drive → Manage members → add the SA email as Viewer or Content Manager.

2c. (Optional) Browser picker bootstrap values

The service-mode google_drive provider can advertise the in-browser Google Picker so users can browse the SA's accessible files instead of pasting URLs. This is purely a client UX upgrade — it requires no new server credentials, no per-user OAuth, and no encryption key. If you skip this step, clients fall back to the URL-paste flow against the same SA.

The Picker SDK runs entirely in the browser and needs three values, all of which Google designs as public/browser-safe — they identify your application but do not authorize anything on their own. The user OAuth handshake happens client-side via Google Identity Services with PKCE; no client_secret is involved or surfaced.

  1. Web OAuth Client ID — APIs & Services → Credentials → Create Credentials → OAuth client ID → Web application.
    • Authorized JavaScript origins: add the origin(s) that host your TMI front-end (e.g. http://localhost:4200 for local dev, https://app.example.com for production). The SDK rejects browser requests from any origin not on this list.
    • Authorized redirect URIs: leave empty. The Picker uses Google Identity Services with PKCE and does not need a redirect target.
    • After creation, copy the Client ID (looks like 1234567890-abc.apps.googleusercontent.com).
    • Discard the Client Secret. It is generated alongside the client id but is not used by this flow and must not be put anywhere TMI or its config files can see — it is not browser-safe and the PKCE flow proves possession without it.
  2. Picker Developer Key — APIs & Services → Credentials → Create Credentials → API key. Edit the new key:
    • Application restrictions: HTTP referrers → add the same front-end origin(s) with a /* suffix (e.g. http://localhost:4200/*, https://app.example.com/*).
    • API restrictions: Restrict key → select Google Picker API only.
    • Make sure the Google Picker API is enabled for the project (APIs & Services → Library → Google Picker API → Enable).
    • Copy the key (looks like AIzaSy...).
  3. GCP Project Number — Cloud Console → IAM & Admin → Settings. Copy the Project number (numeric, e.g. 675196260523). This is not the project id string (e.g. tmi-server).

These three values become picker_config.client_id, picker_config.developer_key, and picker_config.app_id on the google_drive entry of the /config response. The server emits picker_config only when all three are set — partial configuration is treated as unconfigured and the field is omitted.

2d. TMI configuration

YAML (under content_sources):

content_sources:
  google_drive:
    enabled: true
    service_account_email: "tmi-content-gdrive@<project>.iam.gserviceaccount.com"
    credentials_file: "/etc/tmi/google-drive-credentials.json"
    # Optional — enables the in-browser Google Picker UX (Step 2c).
    # All three required to advertise picker_config; partial config is ignored.
    browser_oauth_client_id: "<Web OAuth client id from step 2c.1>"
    picker_developer_key: "<Picker API key from step 2c.2>"
    picker_app_id: "<GCP project number from step 2c.3>"

Or environment variables:

Variable Purpose
TMI_CONTENT_SOURCE_GOOGLE_DRIVE_ENABLED true to enable.
TMI_CONTENT_SOURCE_GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL The SA email from step 2a. Surfaced to users in the share-with-the-bot remediation message.
TMI_CONTENT_SOURCE_GOOGLE_DRIVE_CREDENTIALS_FILE Absolute path to the SA JSON key.
TMI_CONTENT_SOURCE_GOOGLE_DRIVE_BROWSER_OAUTH_CLIENT_ID (Optional) Web OAuth client id from step 2c.1. Surfaced to the browser via /config. Browser-safe.
TMI_CONTENT_SOURCE_GOOGLE_DRIVE_PICKER_DEVELOPER_KEY (Optional) Picker API key from step 2c.2. Surfaced to the browser via /config. Browser-safe.
TMI_CONTENT_SOURCE_GOOGLE_DRIVE_PICKER_APP_ID (Optional) GCP project number from step 2c.3. Surfaced to the browser via /config. Browser-safe.

That's all google_drive needs. There is no OAuth provider entry, no encryption key, and no per-user link flow — even with the picker enabled, the picker UX uses the user's own browser session against Google and the actual file fetch still happens via the service account.

Step 3 — google_workspace (delegated + picker) setup

This is more involved. It uses per-user OAuth tokens, encrypted at rest, plus the Google Picker SDK in the browser.

3a. Create an OAuth 2.0 Client ID

  1. APIs & Services → OAuth consent screen → configure the app.
    • For an internal Google Workspace org, choose Internal.
    • For external testing, choose External and add yourself as a test user.
  2. Add the OAuth scope https://www.googleapis.com/auth/drive.file. TMI deliberately uses the narrow drive.file scope so it can only read files the user explicitly picks.
  3. APIs & Services → Credentials → Create Credentials → OAuth client ID → Web application.
  4. Add an Authorized redirect URI that matches TMI's content callback. For local dev that is http://localhost:8080/oauth2/content_callback. For production, use the public URL of your TMI deployment with the same path.
  5. After creation, copy the Client ID and Client Secret.

3b. Get the Picker API key and App ID

The browser-side Google Picker SDK needs two non-secret values:

  1. Picker Developer Key — in Cloud Console: Credentials → Create Credentials → API key. Restrict it to the Google Picker API and to your TMI front-end's HTTP referrers (e.g. https://app.example.com/*).
  2. Picker App ID — your GCP project number (not the project ID). Find it on Cloud Console → Home → Project info → Project number.

Make sure the Google Picker API is enabled (Step 1 covers this).

3c. Set the content-token encryption key

Per-user tokens are stored AES-256-GCM-encrypted at rest. You need a 32-byte hex-encoded key (64 hex characters):

openssl rand -hex 32

Set it via:

content_token_encryption_key: "<64 hex chars>"

or env: TMI_CONTENT_TOKEN_ENCRYPTION_KEY=<64 hex chars>.

If google_workspace is enabled and this key is missing, the server refuses to start. Rotating this key invalidates all stored content tokens — every user will have to re-link their Google account.

3d. Wire the OAuth provider entry

Under content_oauth.providers, add a google_workspace entry:

content_oauth:
  callback_url: "https://tmi.example.com/oauth2/content_callback"
  allowed_client_callbacks:
    - "https://app.example.com/*"
  providers:
    google_workspace:
      enabled: true
      name: "Google"
      icon: "/static/provider-logos/signin/google.svg"
      client_id: "<from step 3a>"
      client_secret: "<from step 3a>"
      auth_url: "https://accounts.google.com/o/oauth2/v2/auth"
      token_url: "https://oauth2.googleapis.com/token"
      userinfo_url: "https://openidconnect.googleapis.com/v1/userinfo"
      revocation_url: "https://oauth2.googleapis.com/revoke"
      required_scopes:
        - "https://www.googleapis.com/auth/drive.file"
        - "openid"
        - "email"
      extra_authorize_params:
        access_type: "offline"   # required to receive a refresh token
        prompt: "consent"        # forces refresh-token issuance on every (re)link

Notes:

  • access_type=offline is required. Without it, Google does not return a refresh token, and users will have to re-link every hour when their access token expires.
  • prompt=consent is recommended — Google only re-issues a refresh token if the user goes through the consent screen, so without this, re-linking does not produce a new refresh token.
  • The revocation_url is optional but recommended. If set, TMI calls Google's RFC 7009 endpoint when a user unlinks (DELETE /me/content_tokens/google_workspace).

3e. Wire the content source

Under content_sources:

content_sources:
  google_workspace:
    enabled: true
    picker_developer_key: "<API key from step 3b>"
    picker_app_id: "<GCP project number from step 3b>"

Or environment variables:

Variable Purpose
TMI_CONTENT_SOURCE_GOOGLE_WORKSPACE_ENABLED true to enable.
TMI_CONTENT_SOURCE_GOOGLE_WORKSPACE_PICKER_DEVELOPER_KEY Picker API key from step 3b. Surfaced to the browser.
TMI_CONTENT_SOURCE_GOOGLE_WORKSPACE_PICKER_APP_ID GCP project number from step 3b. Surfaced to the browser.

Startup will fail fast if enabled=true but either picker value is empty, or the OAuth provider entry above is missing or disabled. The error message in the log identifies which check tripped.

Step 4 — Verify

Start the server (e.g. make dev-up) and watch the log for:

  • Content source enabled: google_drive — service-account source registered.
  • Content source enabled: google_workspace (delegated, drive.file scope) — delegated source registered.

If you configured the optional browser picker bootstrap for google_drive (Step 2c), confirm the round-trip:

curl -s http://localhost:8080/config | jq '.content_providers[] | select(.id=="google_drive")'

Expect a picker_config object with client_id, developer_key, and app_id. If picker_config is absent, at least one of the three fields is empty — the server emits the object only when all three are set.

For google_workspace, exercise the link flow end-to-end:

POST /me/content_tokens/google_workspace/authorize
     body: {"client_callback": "https://app.example.com/content-linked"}
→ {"authorization_url": "https://accounts.google.com/...", "expires_at": "..."}

The browser navigates the user to authorization_url, they grant consent, Google redirects to /oauth2/content_callback, TMI stores the encrypted token, and the user's browser is redirected to the client_callback URL with status=success.

After a successful link:

  • GET /me/content_tokens lists google_workspace for the calling user.
  • DELETE /me/content_tokens/google_workspace revokes the token at Google (if revocation_url is configured) and removes the row.

Both providers enabled at once

If both google_drive and google_workspace are enabled, the registry is wired so that the delegated google_workspace source is matched before google_drive for the same Drive URLs. In practice:

  • A document attached via the Google Picker (carrying a picker_registration payload tied to the user's linked token) is fetched via google_workspace using the user's own credentials.
  • A document with a plain Drive URL and no picker registration falls through to google_drive and is fetched with the service account, which can only see it if the doc has been shared with the SA email.

This is by design: it lets a deployment serve both per-user content (delegated) and shared corporate content (service account) at the same time without ambiguity.

Picker UI in tmi-ux

The browser-side Google Picker integration lives in tmi-ux (the web client), not in the TMI server. The TMI server's role is to advertise the bootstrap values via /config and (for delegated) to broker the OAuth flow; the SDK initialization and picker UX are client-side.

  • google_drive (service-mode picker) — consumes content_providers[id=google_drive].picker_config from /config. Tracked in tmi-ux issue #671. Server side shipped in tmi#387.
  • google_workspace (delegated picker) — consumes picker_developer_key / picker_app_id plus the picker_registration payload sent to TMI on document attach. Tracked in tmi-ux issue #626.

The TMI server side is complete for both; the operator-facing setup on this page is what the deployment needs in any case.

Local development quick path

For local development without a real Google account, the dev config (config-development.yml) ships a devstub OAuth provider that runs against scripts/oauth-client-callback-stub.py. You can exercise the entire delegated link flow against the stub before plugging in real Google credentials. See Delegated Content Providers for the architecture; Authentication for the OAuth callback stub.

Troubleshooting

Symptom Likely cause
Server refuses to start: content_sources.google_workspace.enabled=true requires content-token encryption key and OAuth provider configuration TMI_CONTENT_TOKEN_ENCRYPTION_KEY is unset, or content_oauth.providers.google_workspace.enabled=true is missing.
Server refuses to start: requires content_oauth.providers.google_workspace.enabled=true The OAuth provider entry exists but is disabled, or the entry is missing entirely. Add it and re-start.
Server refuses to start: requires picker_developer_key and picker_app_id content_sources.google_workspace.picker_developer_key or picker_app_id is empty. Fill in both.
Users land on the callback with error=invalid_state The OAuth state TTL is 10 minutes. The user took too long, or Redis was reset between authorize and callback. Restart the link flow.
Users land on the callback with error=client_callback_not_allowed The client_callback URL the UI supplied is not in content_oauth.allowed_client_callbacks. Add it (supports trailing-* wildcards).
Users have to re-link every hour Refresh token is not being issued. Confirm access_type=offline is in extra_authorize_params, and that the user went through a consent prompt (prompt=consent on the next link).
google_drive returns "file not found" for a document the user can see in their browser The file is not shared with the service-account email. Share it with the SA, or attach it via picker (delegated path) instead.
Users see "account needs reconnecting" after a long idle Stored row is in status=failed_refresh — Google rejected the refresh token. The user reconnects via the link flow.

Related

  • Delegated Content Providers — architecture, token lifecycle, link/unlink API, infra-level troubleshooting.
  • Configuration Reference — full inventory of content_sources.* and content_oauth.* keys.
  • Authentication — user-auth OAuth (a different subsystem under oauth.providers.*; do not conflate with content_oauth.providers.*).
  • Tracking issue: ericfitz/tmi#249 — Confluence and OneDrive content providers (Google Workspace shipped under sub-project 4).
⚠️ **GitHub.com Fallback** ⚠️