Service: S3 Sync Vectorstore - EyevinnOSC/community GitHub Wiki
S3 Sync Vectorstore
S3 Sync Vectorstore (serviceId eyevinn-s3-sync-vectorstore) uploads the contents of an S3-compatible bucket to an OpenAI vector store. In Eyevinn Open Source Cloud (OSC) it is a job service: it runs once to completion rather than as a long-running instance. License: MIT. Source: Eyevinn/s3-sync-vectorstore.
Vector stores in OpenAI augment an Assistant with knowledge from outside its model, such as proprietary product information or user documents. OpenAI parses and chunks the documents, creates and stores the embeddings, and uses vector and keyword search to retrieve relevant content.
What it can do
- Sync files from one or more S3 or S3-compatible buckets (AWS S3, OSC MinIO) into an OpenAI vector store.
- Be run on demand, or automated from a script or a GitHub Actions workflow, for example to refresh a vector store when documentation has been regenerated.
Creating an instance
You need an OpenAI account and API key, an existing OpenAI vector store (its id looks like vs_...), and an S3-compatible bucket with the documents. Store the OpenAI key and the S3 credentials as service secrets with create-service-secret (serviceId eyevinn-s3-sync-vectorstore, for example openaikey, accesskeyid, secretaccesskey).
Create the job with create-service-instance and serviceId: "eyevinn-s3-sync-vectorstore"; the name must be alphanumeric and at most 20 characters.
{
"name": "guide1",
"cmdLineArgs": "s3://mybucket/files/ vs_<vector-store-id>",
"OpenaiApiKey": "{{secrets.openaikey}}",
"AwsRegion": "eu-north-1",
"AwsAccessKeyId": "{{secrets.accesskeyid}}",
"AwsSecretAccessKey": "{{secrets.secretaccesskey}}"
}
Follow the job with get-instance-status and get-logs-for-instance. When it has completed, check in OpenAI that the files are in the vector store, then add the vector store as a file search tool on your assistant. Remove a finished job with delete-service-instance before reusing its name.
Options
| Option | Required | Type | Meaning |
|---|---|---|---|
| name | Yes | string | Name of the job. Alphanumeric, at most 20 characters. |
| cmdLineArgs | Yes | string | The command-line arguments: the source S3 URL followed by the OpenAI vector store id, separated by a space, for example s3://source/files vs_3434533213. Several source locations can be comma-separated, for example s3://source/files/,s3://source2/files/. |
| OpenaiApiKey | Yes | string, sensitive | OpenAI API key (OPENAI_API_KEY). Store it as a secret, for example {{secrets.openaikey}}. |
| Purpose | No | string | Purpose of the files in the vector store (OPENAI_PURPOSE). The OSC guide states the default is assistants; see the OpenAI documentation for valid values. |
| AwsRegion | No | string | Region of the S3 bucket if it is on AWS (AWS_REGION), for example eu-north-1. |
| S3Endpoint | No | string | Endpoint URL for buckets in OSC or from non-AWS vendors (S3_ENDPOINT). Not needed for standard AWS S3. |
| AwsAccessKeyId | No | string, sensitive | Access key id for the bucket (AWS_ACCESS_KEY_ID). |
| AwsSecretAccessKey | No | string, sensitive | Secret access key for the bucket (AWS_SECRET_ACCESS_KEY). |
Sensitive options should be stored as service secrets and passed as {{secrets.NAME}}, which only works as the entire field value.
Using the instance
CLI
% export OSC_ACCESS_TOKEN=<your-osc-pat>
% npx @osaas/cli@latest create eyevinn-s3-sync-vectorstore guide2 \
-o cmdLineArgs="s3://origin-osaas-client-jsdocs/ vs_67c7f18585d481918824cd12b135870c" \
-o OpenaiApiKey="{{secrets.openaidevkey}}" \
-o AwsRegion="eu-north-1" \
-o AwsAccessKeyId="{{secrets.accesskeyid}}" \
-o AwsSecretAccessKey="{{secrets.secretaccesskey}}"
GitHub Actions
rag:
needs: build_docs
runs-on: ubuntu-latest
steps:
- uses: actions/setup-node@v3
with:
node-version: '18.x'
registry-url: https://registry.npmjs.org/
- name: Create OSC job to sync vector store
run: |
npx -y @osaas/cli@latest remove eyevinn-s3-sync-vectorstore ghsync
npx -y @osaas/cli@latest create eyevinn-s3-sync-vectorstore ghsync \
-o cmdLineArgs="s3://origin-osaas-client-jsdocs/ vs_67c84ced60c48191affdb63969dd5494" \
-o OpenaiApiKey="{{secrets.openaikey}}" \
-o AwsRegion="eu-north-1" \
-o AwsAccessKeyId="{{secrets.accesskeyid}}" \
-o AwsSecretAccessKey="{{secrets.secretaccesskey}}"
env:
OSC_ACCESS_TOKEN: ${{ secrets.OSC_ACCESS_TOKEN }}