Service: S3 Sync Vectorstore - EyevinnOSC/community GitHub Wiki

S3 Sync Vectorstore

S3 Sync Vectorstore (serviceId eyevinn-s3-sync-vectorstore) uploads the contents of an S3-compatible bucket to an OpenAI vector store. In Eyevinn Open Source Cloud (OSC) it is a job service: it runs once to completion rather than as a long-running instance. License: MIT. Source: Eyevinn/s3-sync-vectorstore.

Vector stores in OpenAI augment an Assistant with knowledge from outside its model, such as proprietary product information or user documents. OpenAI parses and chunks the documents, creates and stores the embeddings, and uses vector and keyword search to retrieve relevant content.

What it can do

  • Sync files from one or more S3 or S3-compatible buckets (AWS S3, OSC MinIO) into an OpenAI vector store.
  • Be run on demand, or automated from a script or a GitHub Actions workflow, for example to refresh a vector store when documentation has been regenerated.

Creating an instance

You need an OpenAI account and API key, an existing OpenAI vector store (its id looks like vs_...), and an S3-compatible bucket with the documents. Store the OpenAI key and the S3 credentials as service secrets with create-service-secret (serviceId eyevinn-s3-sync-vectorstore, for example openaikey, accesskeyid, secretaccesskey).

Create the job with create-service-instance and serviceId: "eyevinn-s3-sync-vectorstore"; the name must be alphanumeric and at most 20 characters.

{
  "name": "guide1",
  "cmdLineArgs": "s3://mybucket/files/ vs_<vector-store-id>",
  "OpenaiApiKey": "{{secrets.openaikey}}",
  "AwsRegion": "eu-north-1",
  "AwsAccessKeyId": "{{secrets.accesskeyid}}",
  "AwsSecretAccessKey": "{{secrets.secretaccesskey}}"
}

Follow the job with get-instance-status and get-logs-for-instance. When it has completed, check in OpenAI that the files are in the vector store, then add the vector store as a file search tool on your assistant. Remove a finished job with delete-service-instance before reusing its name.

Options

Option Required Type Meaning
name Yes string Name of the job. Alphanumeric, at most 20 characters.
cmdLineArgs Yes string The command-line arguments: the source S3 URL followed by the OpenAI vector store id, separated by a space, for example s3://source/files vs_3434533213. Several source locations can be comma-separated, for example s3://source/files/,s3://source2/files/.
OpenaiApiKey Yes string, sensitive OpenAI API key (OPENAI_API_KEY). Store it as a secret, for example {{secrets.openaikey}}.
Purpose No string Purpose of the files in the vector store (OPENAI_PURPOSE). The OSC guide states the default is assistants; see the OpenAI documentation for valid values.
AwsRegion No string Region of the S3 bucket if it is on AWS (AWS_REGION), for example eu-north-1.
S3Endpoint No string Endpoint URL for buckets in OSC or from non-AWS vendors (S3_ENDPOINT). Not needed for standard AWS S3.
AwsAccessKeyId No string, sensitive Access key id for the bucket (AWS_ACCESS_KEY_ID).
AwsSecretAccessKey No string, sensitive Secret access key for the bucket (AWS_SECRET_ACCESS_KEY).

Sensitive options should be stored as service secrets and passed as {{secrets.NAME}}, which only works as the entire field value.

Using the instance

CLI

% export OSC_ACCESS_TOKEN=<your-osc-pat>
% npx @osaas/cli@latest create eyevinn-s3-sync-vectorstore guide2 \
  -o cmdLineArgs="s3://origin-osaas-client-jsdocs/ vs_67c7f18585d481918824cd12b135870c" \
  -o OpenaiApiKey="{{secrets.openaidevkey}}" \
  -o AwsRegion="eu-north-1" \
  -o AwsAccessKeyId="{{secrets.accesskeyid}}" \
  -o AwsSecretAccessKey="{{secrets.secretaccesskey}}"

GitHub Actions

  rag:
    needs: build_docs
    runs-on: ubuntu-latest

    steps:
      - uses: actions/setup-node@v3
        with:
          node-version: '18.x'
          registry-url: https://registry.npmjs.org/
      - name: Create OSC job to sync vector store
        run: |
          npx -y @osaas/cli@latest remove eyevinn-s3-sync-vectorstore ghsync
          npx -y @osaas/cli@latest create eyevinn-s3-sync-vectorstore ghsync \
            -o cmdLineArgs="s3://origin-osaas-client-jsdocs/ vs_67c84ced60c48191affdb63969dd5494" \
            -o OpenaiApiKey="{{secrets.openaikey}}" \
            -o AwsRegion="eu-north-1" \
            -o AwsAccessKeyId="{{secrets.accesskeyid}}" \
            -o AwsSecretAccessKey="{{secrets.secretaccesskey}}"
        env:
          OSC_ACCESS_TOKEN: ${{ secrets.OSC_ACCESS_TOKEN }}

Resources