OpenCrawling is built around a connector-based architecture. Every step of the pipeline — ingesting content from a source, transforming it via embeddings, and writing it to a destination — is handled by a dedicated connector implementing a well-defined Java interface.

RepositoryConnector (in oc-core):

public non-sealed interface RepositoryConnector extends Connector {
    Flux<RepositoryDocument> scan(String basePath);
}

OutputConnector (in oc-core):

public non-sealed interface OutputConnector extends Connector {
    Mono<Void> send(RepositoryDocument document);
}

Both extend Connector which provides getName(), connect(), and disconnect() lifecycle methods.

Repository connectors are responsible for discovering and streaming documents from content sources into the pipeline.

Module: oc-filesystem-repository-connector
Class: FileSystemRepositoryConnector
Spring Bean: @Primary — selected by default when no other connector is specified

Recursively scans a local filesystem directory using Java NIO and emits each regular file as a RepositoryDocument. Subdirectory traversal is parallelised using Java 25 Structured Task Scope for high-throughput I/O on deep directory trees.

No external configuration properties are required. The path is provided at runtime via the scan(String basePath) call (e.g., from the Admin UI job form or the crawler SCAN_PATH environment variable).

Set the connector type to filesystem in the Admin UI or via the environment variable:

# docker-compose-decoupled.yml
environment:
  CONNECTOR_TYPE: filesystem
  SCAN_PATH: /data

• Files are streamed; content is not loaded into memory until processed.
• The uri is a file:// URI (e.g. file:///data/report.pdf), so no Claim Check Pattern is used — the IngestionConsumer reads directly from the local filesystem path.

Module: oc-alfresco-repository-connector
Class: AlfrescoRepositoryConnector
Security Mapper: AlfrescoSecurityMapper
Spring Bean: @Component (activated when CONNECTOR_TYPE=alfresco or spring.opencrawling.connector.type=alfresco)
Dedicated Guide: Alfresco Content Services (ACS) Connector

Connects to an Alfresco Content Services (Community or Enterprise) instance via the native Alfresco Core REST API v1 and Alfresco Search REST API v1. Supports dual-mode ingestion:

1. Hierarchical Folder Tree Traversal: Scans from root (-root-), a specific folder path (e.g. /Company Home/Sites), a Node UUID, or a site document library (siteId), with options to toggle subfolder recursion (includeSubfolders) and exclude specific folders (excludedFolders).
2. Search API Query Ingestion: Executes declarative AFTS (Alfresco Full Text Search), CMIS SQL, or Lucene queries against /public/search/versions/1/search.

Features zero-trust ACL mapping via AlfrescoSecurityMapper (extracting inherited and locally set permissions for users and GROUP_* authorities), binary content stream size thresholds (maxContentSizeBytes), and metadata-only indexing (includeContentStream=false).

oc-crawler:
  environment:
    CONNECTOR_TYPE: alfresco
    SCAN_PATH: /Company Home
    ALFRESCO_URL: http://alfresco:8080/alfresco/api/-default-/public/alfresco/versions/1
    ALFRESCO_USERNAME: admin
    ALFRESCO_PASSWORD: admin
    ALFRESCO_CRAWL_MODE: folder
    ALFRESCO_INCLUDE_SUBFOLDERS: "true"
    ALFRESCO_INCLUDE_ACLS: "true"

Because Alfresco documents are fetched from a remote HTTP server, the connector downloads the content stream and saves it to a local shared directory (/data/claims/) before publishing the reference to Kafka. This decouples the slow HTTP download from the fast Kafka publish step.

After a crawl job completes, verify ingestion with:

-- Total chunks stored for Alfresco documents
SELECT count(*) FROM vector_store_1024;

-- Inspect content and source URI of stored chunks
SELECT id,
       LEFT(content, 200)     AS preview,
       metadata->>'name'      AS filename,
       metadata->>'mimeType'  AS mime_type,
       metadata->>'uri'       AS source_uri
FROM vector_store_1024
ORDER BY id
LIMIT 10;

-- Semantic similarity search
SELECT id,
       LEFT(content, 300)    AS preview,
       metadata->>'name'     AS filename,
       1 - (embedding <=> (
           SELECT embedding FROM vector_store_1024 LIMIT 1
       ))                    AS similarity
FROM vector_store_1024
ORDER BY similarity DESC
LIMIT 5;

Module: oc-iceberg-repository-connector
Class: IcebergRepositoryConnector
Spring Bean: @Component (activated when CONNECTOR_TYPE=iceberg)

Connects to an Apache Iceberg table catalog and scans structured data tables, emitting each Iceberg Record as a JSON-serialized RepositoryDocument. Supports REST, Hive Metastore, Hadoop, AWS Glue, and in-memory catalog backends. File scan tasks are parallelized using Java 25 Structured Task Scope for high-throughput Parquet file reading.

Each record is serialized to JSON and treated as a document, making Iceberg data lakes a first-class ingestion source for RAG pipelines. This connector reads Parquet data files directly from the catalog's underlying storage (local FS, S3, GCS, ADLS, etc.).

Module: oc-flowable-repository-connector
Class: FlowableRepositoryConnector
Security Mapper: FlowableSecurityMapper
Spring Bean: @Component (activated when CONNECTOR_TYPE=flowable or spring.opencrawling.connector.type=flowable)
Dedicated Guide: Flowable Repository Connector

Connects to a Flowable BPMN Engine REST service (/history/historic-process-instances) to crawl completed and active historic workflow executions, structure process details, extract BPMN variables (/history/historic-variable-instances), and resolve runtime/historic identity links (/history/historic-process-instances/{id}/identitylinks).

Process instances are emitted as standard RepositoryDocument instances containing structured JSON summaries (definition key, business key, start/end timestamps, duration, and user ID), entity reference links, and fine-grained SecurityConfig permissions.

The connector supports two zero-trust architectural models:

1. Collocated DB Pattern (pgvector on Flowable PostgreSQL): Entity references (flowable_proc_inst_id, flowable_scope_id, flowable_scope_type) are stamped to allow query-time SQL EXISTS subqueries against native ACT_RU_IDENTITYLINK and ACT_HI_IDENTITYLINK tables, eliminating distributed transaction lag without chunk-level row multiplication.
2. Distributed DB Pattern (External Vector DBs): Extracts user and candidate group identity links into standard OIS security_allowed_read metadata fields for pre-filtering in Qdrant, Milvus, OpenSearch, and Vespa.

Batch scanning and variable resolution are parallelized using Java 25 Structured Task Scope and Virtual Threads (Thread.ofVirtual()).

{
  "id": "proc-1234",
  "uri": "flowable://process-instances/proc-1234",
  "action": "UPSERT",
  "metadata": {
    "mimeType": ["application/json"],
    "processDefinitionId": ["invoice-process:2:98765"],
    "processDefinitionKey": ["invoice-process"],
    "businessKey": ["INV-2026-001"],
    "startUserId": ["finance_agent_1"],
    "startTime": ["2026-07-23T08:00:00Z"],
    "endTime": ["2026-07-23T08:15:00Z"],
    "durationInMillis": ["900000"],
    "flowable_proc_inst_id": ["proc-1234"],
    "flowable_process_instance_id": ["proc-1234"],
    "flowable_scope_id": ["proc-1234"],
    "flowable_scope_type": ["processInstance"],
    "flowable_identity_users": ["finance_agent_1"],
    "flowable_identity_groups": ["finance_managers"],
    "flowable_var_totalAmount": ["250.50"],
    "flowable_var_customerName": ["ACME Corp"]
  },
  "security": {
    "inheritanceEnabled": false,
    "permissions": [
      {"identity": "finance_agent_1", "identityType": "user", "access": "read"},
      {"identity": "finance_managers", "identityType": "group", "access": "read"}
    ]
  }
}

Module: oc-camunda-repository-connector
Class: CamundaRepositoryConnector
Security Mapper: CamundaSecurityMapper
Spring Bean: @Component (activated when CONNECTOR_TYPE=camunda or spring.opencrawling.connector.type=camunda)
Dedicated Guide: Camunda Repository Connector

Connects to a Camunda Platform 7 REST Engine (/engine-rest/history/process-instance) to crawl historic workflow executions, process instance details, historical BPMN process variables (/history/variable-instance), and historic identity link authorizations (/history/identity-link-log).

Process instances are emitted as standard RepositoryDocument instances containing structured JSON summaries (definition key, business key, start/end timestamps, duration, and start user ID), entity reference links, and fine-grained SecurityConfig permissions.

The connector supports two zero-trust architectural models:

1. Collocated DB Pattern (pgvector on Camunda PostgreSQL): Entity references (camunda_proc_inst_id, camunda_scope_id, camunda_scope_type) are stamped to allow query-time SQL EXISTS subqueries against native ACT_RU_IDENTITYLINK and ACT_HI_IDENTITYLINK tables, eliminating distributed transaction lag without chunk-level row multiplication.
2. Distributed DB Pattern (External Vector DBs): Extracts user and candidate group identity links into standard OIS security_allowed_read metadata fields for pre-filtering in Qdrant, Milvus, OpenSearch, and Vespa.

Batch scanning and variable resolution are parallelized using Java 25 Structured Task Scope and Virtual Threads (Thread.ofVirtual()).

{
  "id": "proc-9988",
  "uri": "camunda://process-instances/proc-9988",
  "action": "UPSERT",
  "metadata": {
    "mimeType": ["application/json"],
    "processDefinitionId": ["order-process:1:1001"],
    "processDefinitionKey": ["order-process"],
    "businessKey": ["ORD-2026-999"],
    "startUserId": ["sales_rep_42"],
    "startTime": ["2026-07-24T10:00:00Z"],
    "endTime": ["2026-07-24T10:05:00Z"],
    "durationInMillis": ["300000"],
    "tenantId": ["tenant-alpha"],
    "camunda_proc_inst_id": ["proc-9988"],
    "camunda_process_instance_id": ["proc-9988"],
    "camunda_scope_id": ["proc-9988"],
    "camunda_scope_type": ["processInstance"],
    "camunda_identity_users": ["sales_rep_42", "approver_1"],
    "camunda_identity_groups": ["finance-managers"],
    "camunda_var_orderAmount": ["1500.00"],
    "camunda_var_approvalStatus": ["APPROVED"]
  },
  "security": {
    "inheritanceEnabled": false,
    "permissions": [
      {"identity": "sales_rep_42", "identityType": "user", "access": "read"},
      {"identity": "approver_1", "identityType": "user", "access": "read"},
      {"identity": "finance-managers", "identityType": "group", "access": "read"}
    ]
  }
}

The basePath argument to scan() is an Iceberg table identifier in the format <namespace>.<table>:

The document ID is resolved in order of priority:

1. The column specified in idColumn config property
2. Iceberg table's declared identifier fields
3. SHA-256 hash of the JSON record content (fallback)

Use the dedicated compose file provided in the oc-iceberg-repository-connector module to spin up a local Iceberg REST catalog backed by MinIO (S3-compatible object storage):

# oc-iceberg-repository-connector/docker-compose.yml
services:
  minio:
    image: minio/minio:RELEASE.2024-05-10T01-39-19Z
    container_name: minio-iceberg
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: password
    ports:
      - "9000:9000"
      - "9001:9001"
    command: ["server", "/data", "--console-address", ":9001"]

  iceberg-rest:
    image: apache/iceberg-rest-fixture:1.5.0
    container_name: iceberg-rest
    ports:
      - "8181:8181"
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000
      - CATALOG_S3_PATH_STYLE_ACCESS=true
    depends_on:
      - minio

Then configure the crawler to use this catalog:

oc-crawler:
  environment:
    CONNECTOR_TYPE: iceberg
    SCAN_PATH: analytics.events
    ICEBERG_CATALOG_TYPE: rest
    ICEBERG_CATALOG_URI: http://iceberg-rest:8181
    ICEBERG_WAREHOUSE: s3://warehouse/

• The connector does not use the Claim Check Pattern — Iceberg records are serialized to JSON in-process and streamed directly as RepositoryDocument objects. No shared volume is required.
• The uri field uses an iceberg:// scheme that identifies the source table and record ID.
• For timestamp-based last-modified detection, the connector inspects well-known column names: last_modified, updated_at, timestamp, and date.
• AWS Glue catalog requires the iceberg-aws dependency on the classpath and AWS credentials configured via environment variables or instance profile.

Module: oc-stormcrawler-repository-connector (with oc-stormcrawler-bolt)
Class: StormCrawlerRepositoryConnector
Spring Bean: @Component (activated when CONNECTOR_TYPE=stormcrawler)
Dedicated Guide: Apache StormCrawler Repository Connector & OpenCrawling Bolt

Orchestrates, monitors, and streams distributed web crawls using Apache StormCrawler 3.7.0 running on Apache Storm 3.1.0 clusters and Java 25.

The connector interacts with the Apache Storm Nimbus REST API to verify cluster health (/api/v1/cluster/summary) and manage topology states (/api/v1/topology/summary). For distributed worker processing, the accompanying OpenCrawlingBolt intercepts parsed HTML content and deletion status streams (HTTP 404/410), normalising them into Open Ingestion Standard (OIS) UPSERT and DELETE payloads dispatched to OpenCrawling.

High-throughput seed scanning and URL resolution leverage Java 25 Virtual Threads and StructuredTaskScope.

# docker-compose-decoupled-with-stormcrawler.yml
services:
  oc-crawler:
    environment:
      CONNECTOR_TYPE: stormcrawler
      STORMCRAWLER_NIMBUS_HOST: storm-nimbus
      STORMCRAWLER_NIMBUS_REST_URL: http://storm-ui:8080
      STORMCRAWLER_TOPOLOGY_NAME: opencrawling-web-crawler
      STORMCRAWLER_SEEDS: https://docs.opencrawling.org

Module: oc-aps-repository-connector
Class: ApsRepositoryConnector (with ApsMcpClient)
Spring Bean: @Component (activated when CONNECTOR_TYPE=aps)
Dedicated Guide: Alfresco Process Services (APS) Repository Connector

Connects to Alfresco Process Services (APS) 26.2 via the Enterprise REST API (/activiti-app/api/enterprise) to crawl historic and active process executions (/historic-process-instances/query), dynamically extract all process variables (/historic-process-instances/{id}/variables), and evaluate user tasks and candidate group authorizations (/tasks/query).

In addition to workflow execution metrics, the connector features Workflow Attachment Ingestion: it discovers documents and text attachments attached at the process instance level or task level (/content/{contentId}/raw), streams them into the Claim-Check store, extracts text with Apache Tika, and links them to the parent workflow with inherited variables and Zero-Trust Access Control Lists (ACLs).

The connector supports two zero-trust architectural models:

1. Collocated DB Pattern (pgvector on APS PostgreSQL): Entity references (aps_proc_inst_id, aps_scope_id, aps_scope_type) are stamped to allow query-time SQL EXISTS subqueries against native ACT_RU_IDENTITYLINK and ACT_HI_IDENTITYLINK tables, eliminating distributed transaction lag without chunk-level row multiplication.
2. Distributed DB Pattern (External Vector DBs): Extracts user and candidate group identity links into standard OIS security_allowed_read metadata fields for pre-filtering in Qdrant, Milvus, OpenSearch, and Vespa.

The connector also provides an ApsMcpClient to dynamically discover deployed workflow definitions and schema metadata via the Model Context Protocol (MCP). Batch processing is parallelized using Java 25 Virtual Threads and StructuredTaskScope.

oc-crawler:
  environment:
    CONNECTOR_TYPE: aps
    SCAN_PATH: /
    APS_URL: http://aps:8080/activiti-app/api/enterprise
    APS_USERNAME: [email protected]
    APS_PASSWORD: admin
    APS_INCLUDE_VARIABLES: "true"
    APS_INCLUDE_TASKS: "true"
    APS_INCLUDE_ATTACHMENTS: "true"

Module: oc-cmis-repository-connector
Class: CmisRepositoryConnector
Spring Bean: @Component (activated when CONNECTOR_TYPE=cmis or spring.opencrawling.connector.type=cmis)
Dedicated Guide: OASIS CMIS Repository Connector

Connects to any OASIS CMIS 1.0 / 1.1 compliant Enterprise Content Management (ECM) repository — including Alfresco Content Services, OpenText Documentum, IBM FileNet P8, Apache Chemistry InMemory / FileShare, and Nuxeo — using the high-performance CMIS 1.1 Browser Binding (JSON over HTTP/REST).

Implemented natively with Java 25 HttpClient (zero legacy Apache Chemistry dependencies), the connector parallelizes folder hierarchy scanning, metadata enrichment, and binary content extraction using Java 25 Virtual Threads (StructuredTaskScope).

It supports:

• Multiple Crawl Modes: Recursive folder traversal (folder), CMIS-SQL relational queries (query), and incremental change log synchronization.
• Versioning Policies: latest_major (official releases), latest (including minor revisions), or all (complete history).
• Aspects & Secondary Types: Full extraction of cmis:secondaryObjectTypeIds and arbitrary custom metadata namespaces.
• Zero-Trust Security: Maps CMIS Access Control Lists (ACLs) and ACEs into OIS SecurityConfig permission rules.
• Claim Check Stream Pattern: Routes binary streams directly to local disk, Apache Ozone, or AWS S3, passing lightweight URI references to Kafka.

oc-crawler:
  image: opencrawling/oc-runtime:latest
  environment:
    CONNECTOR_TYPE: cmis
    SCAN_PATH: /Company Home
    CMIS_ENDPOINT_URL: http://alfresco:8080/alfresco/api/-default-/public/cmis/versions/1.1/browser
    CMIS_REPOSITORY_ID: -default-
    CMIS_USERNAME: admin
    CMIS_PASSWORD: admin
    CMIS_CRAWL_MODE: folder
    CMIS_INCLUDE_SUBFOLDERS: "true"
    CMIS_VERSIONS_MODE: latest_major
    CMIS_INCLUDE_ACLS: "true"
    CMIS_INCLUDE_CONTENT_STREAM: "true"

Module: oc-jdbc-repository-connector
Class: JdbcRepositoryConnector
Security Mapper: JdbcSecurityMapper
Crawl Mode Enum: JdbcCrawlMode
Spring Bean: @Component (activated when CONNECTOR_TYPE=jdbc or spring.opencrawling.connector.type=jdbc)
Dedicated Guide: Relational Database (JDBC) Connector

Connects natively to any JDBC-compliant Relational Database Management System (RDBMS) — including PostgreSQL, MySQL, MariaDB, Oracle Database, Microsoft SQL Server, IBM DB2, SQLite, DuckDB, H2, and Snowflake — via standard JDBC 4.2+ drivers.

The connector supports dual-path ingestion:

1. Tabular RAG & Auto-Narrativization: Ingests relational rows from tables, views, or arbitrary SQL queries, introspects schema metadata via getSchema(basePath) to empower Spring AI TemplateGenerationCopilot, and renders rows into cohesive natural-language narrative text (or deterministic Markdown key-values) for embedding.
2. Binary BLOB / CLOB Streaming via Claim Check: Directly streams database binary columns (BLOB, BYTEA, RAW, CLOB) into the active ClaimCheckStore (Local disk, Apache Ozone, AWS S3) with automated magic byte sniffing (PDF, PNG, JPEG, ZIP, GIF, WEBP) and Apache Tika text/OCR extraction without JVM heap exhaustion.

Built on Java 25 Virtual Threads (StructuredTaskScope) and HikariCP connection pooling, it features multi-table concurrent scanning, engine-specific streaming cursors (PostgreSQL autocommit disabled, MySQL streaming fetch), stateful High-Water Mark (HWM) incremental delta crawling, soft-delete tombstone emission (DocumentAction.DELETE), and zero-trust column-to-ACL security mapping (JdbcSecurityMapper).

oc-crawler:
  image: opencrawling/oc-runtime:latest
  environment:
    CONNECTOR_TYPE: jdbc
    SCAN_PATH: support_tickets
    JDBC_URL: jdbc:postgresql://postgres-source:5432/crm_prod
    JDBC_DRIVER_CLASS_NAME: org.postgresql.Driver
    JDBC_USERNAME: crm_reader
    JDBC_PASSWORD: crm_password
    JDBC_TABLE_NAME: support_tickets
    JDBC_PRIMARY_KEY_COLUMNS: id
    JDBC_TITLE_COLUMN: title
    JDBC_SECURITY_ENABLED: "true"
    JDBC_USER_COLUMNS: owner_id
    JDBC_GROUP_COLUMNS: department_id
    JDBC_SOFT_DELETE_ENABLED: "true"
    JDBC_SOFT_DELETE_COLUMN: is_deleted
    JDBC_SOFT_DELETE_VALUE: "true"

Output connectors receive a RepositoryDocument and persist it directly (bypassing Kafka — used in the direct/standalone execution mode).

Module: oc-vector-output-connector
Class: VectorOutputConnector
Spring Bean: @Component

Parses document content using Apache Tika (with full parser support for PDF, Word, PowerPoint, Excel, HTML, plain text, and more), splits the extracted text into overlapping chunks, generates embeddings by calling the configured embedding model, and writes the resulting vectors to PGVector (PostgreSQL with the pgvector extension).

This connector is used in the direct output mode (where the crawler and vectorisation happen in the same JVM process). In the decoupled mode, the equivalent pipeline is executed across three separate microservices: oc-ingestion-consumer, oc-embedding-service, and oc-runtime (VectorStoreWriterConsumer).

The connector maintains four separate tables to support different embedding model dimension sizes simultaneously:

The correct table is selected automatically based on the length of the embedding vector returned by the model.

Apache Tika's AutoDetectParser handles format detection and extraction. If parsing fails for a specific file (e.g., a FreeMarker .ftl template mistakenly detected as RSS/XML), the connector logs a warning and falls back to reading the file as UTF-8 plain text.

Module: oc-milvus-output-connector
Class: MilvusOutputConnector
Spring Bean: @Component (activated when spring.opencrawling.output.type=milvus)

Persists precomputed vector embeddings and text chunks directly into a Milvus vector database collection. It supports standard index types (e.g., HNSW), custom metric types (e.g., COSINE), and user token authentication.

In the decoupled architecture, MilvusStoreWriterConsumer consumes embedded chunks from the Kafka opencrawling-embedded topic and saves them using this connector.

Module: oc-opensearch2-output-connector
Class: OpenSearch2OutputConnector
Spring Bean: @Component (activated when spring.opencrawling.output.type=opensearch2)

Persists precomputed vector embeddings and text chunks directly into an OpenSearch 2.x instance or cluster. On startup, the connector checks for the existence of the target index and automatically initializes it with k-NN HNSW settings and cosine similarity. It relocates internal org.opensearch classes under a shaded packaging boundary (org.opencrawling.shaded.opensearch2) to avoid classpath namespace conflicts with OpenSearch 3.x.

It also contains built-in connection retries (up to 10 attempts, sleeping 3 seconds between attempts) to gracefully handle container orchestration startup delays.

Module: oc-opensearch3-output-connector
Class: OpenSearch3OutputConnector
Spring Bean: @Component (activated when spring.opencrawling.output.type=opensearch3)

Persists precomputed vector embeddings and text chunks directly into an OpenSearch 3.x instance. Similar to the 2.x connector, it automatically checks and boots up the target index using k-NN settings on startup. It includes built-in connection retries (up to 10 attempts, sleeping 3 seconds between attempts) to gracefully wait for OpenSearch containers to become healthy during multi-container orchestration.

Module: oc-qdrant-output-connector
Class: QdrantOutputConnector
Spring Bean: @Component (activated when spring.opencrawling.output.type=qdrant)

Persists precomputed vector embeddings and text chunks directly into a Qdrant vector database collection (self-hosted or Qdrant Cloud). It utilizes Qdrant's high-performance gRPC Java Client (io.qdrant:client) for low-latency batch upserts of PointStruct objects.

Upon initialization via QdrantCollectionInitializer, it automatically verifies or provisions the target Qdrant collection with configured vector parameters (dimensions, Cosine/Dot/Euclid distance metrics) and creates KEYWORD payload indexes for security ACL fields (security_allowed_read, security_denied_read). This allows sub-millisecond ACL pre-filtering during similarity search in McpVectorServer.

In the decoupled architecture, QdrantStoreWriterConsumer consumes embedded chunks from the Kafka opencrawling-embedded topic and persists them into Qdrant.

Module: oc-vespa-output-connector
Class: VespaOutputConnector
Spring Bean: @Component (activated when spring.opencrawling.output.type=vespa)

Persists precomputed vector embeddings and text chunks directly into Vespa, Yahoo's open-source big data serving engine combining ANN vector search with BM25 lexical ranking.

Document chunks and embeddings are fed using the official com.yahoo.vespa:vespa-feed-client HTTP/2 client into Vespa document types with full support for Open Ingestion Standard (OIS) ACL pre-filtering, dynamic multi-dimension vector routing, mTLS security, and Admin UI Model Insights.

1. Document/v1 feeding via official HTTP/2 client: Both VespaOutputConnector.send() (direct SPI path) and VespaStoreWriterConsumer (decoupled Kafka writer for opencrawling-embedded) stream document chunks using vespa-feed-client, which manages HTTP/2 multiplexed connection pools, dynamic throttling, and automatic retries for transient 429/503 responses.
2. Dynamic Multi-Dimension Routing: VespaDocumentMapper.resolveDocumentType() inspects each chunk's actual vector dimension and dynamically routes it to a dimension-specific Vespa document type:
   • 384 dimensions $\rightarrow$ opencrawling_chunk_384 (e.g. all-minilm)
   • 768 dimensions $\rightarrow$ opencrawling_chunk_768 (e.g. nomic-embed-text)
   • 1024 dimensions $\rightarrow$ opencrawling_chunk_1024 (e.g. mxbai-embed-large)
   • Any other dimension $\rightarrow$ Fallback to the configured default document type (opencrawling_chunk). This allows mixing embedding models across jobs without changing configuration or redeploying schemas.
3. ACL-Aware Security Indexing: Each document carries security_allowed_read, security_denied_read, and security_inheritance fields derived from the OIS security model, enabling exact permission filtering.
4. Structured Schema Safety: Dynamic metadata fields are consolidated into a single metadata_json blob to conform strictly to Vespa's strongly typed schema requirements.
5. Admin UI Model Insights & Deployment: Backed by /api/vespa/* REST endpoints (VespaInsightsService / VespaInsightsController), providing live engine health (/api/vespa/health), per-document-type chunk counts (/api/vespa/document-counts), hybrid/BM25 query testing (/api/vespa/query), and one-click bundled schema deployment (/api/vespa/deploy/bundled).
6. Vespa-Native Secure MCP Server: VespaMcpVectorServer exposes vespaSecureVectorSearch (hybrid BM25 + vector similarity ranking), vespaGetDocumentContent, and vespaListAccessibleSources over SSE/HTTP, pushing ACL security_denied_read checks into Vespa YQL queries while maintaining Java-level OIS re-checking.

All properties are bound under spring.opencrawling.output.vespa.*:

{
  "fields": {
    "chunk_id": "doc-123_a1b2c3",
    "text": "Enterprise search chunk content...",
    "uri": "file:///data/report.pdf",
    "acl": "finance",
    "lastModified": "2026-08-01T10:00:00Z",
    "security_inheritance": true,
    "security_allowed_read": ["user1", "finance-group"],
    "security_denied_read": ["external-contractor"],
    "metadata_json": "{\"title\":[\"Q3 Financial Report\"],\"mimeType\":[\"application/pdf\"]}",
    "embedding": { "values": [0.0123, -0.0456, 0.0891] }
  }
}

# Spin up Vespa with bundled schema auto-deploy via Docker Compose
docker compose -f oc-vespa-output-connector/docker/docker-compose.yml up -d

# Standalone connector smoke test (feeds chunks & asserts dynamic routing + ACLs)
./scripts/test-vespa-connector.sh

# Decoupled Kafka pipeline test with Vespa
./scripts/test-vespa-decoupled.sh

Module: oc-solr-output-connector
Class: SolrOutputConnector
Spring Bean: @Component (activated when spring.opencrawling.output.type=solr)

Persists precomputed vector embeddings and document text chunks into Apache Solr (v10.x / v9.x) using SolrJ 10 native JDK HTTP Client (HttpJdkSolrClient for standalone, CloudSolrClient for SolrCloud mode).

Document chunks are indexed using Solr's solr.DenseVectorField with customizable HNSW graph parameters (hnswMaxConnections, hnswBeamWidth, similarityFunction). It features dynamic REST schema auto-provisioning (SolrCollectionInitializer), Solr 10 vector quantization (scalar, binary), efSearch query tuning, OIS zero-trust ACL security fields, and decoupled Kafka writer consumer execution (SolrStoreWriterConsumer).

For full configuration reference, schema details, and queries, see the Apache Solr Output Connector Wiki Page.

All properties are bound under spring.opencrawling.output.solr.*:

# Spin up standalone Solr 10.x container via Docker Compose
docker compose -f oc-solr-output-connector/docker/docker-compose.yml up -d

# Decoupled Kafka pipeline test with Solr 10
./scripts/test-solr-decoupled.sh

Module: oc-luxir-output-connector
Class: LuxirOutputConnector
Spring Bean: @Component (activated when spring.opencrawling.output.type=luxir)

Persists precomputed vector embeddings, document text chunks, and metadata into Luxir using its native HTTP JSON API.

Document chunks are indexed using Luxir's dynamic schema type conventions (_t, _s, _ss, _v, _vs) and dense vector fields with customizable metrics (cosine, l2, ip). Zero-Trust ACL inheritance is mapped to string (security_inheritance_s: "true"/"false") for seamless dynamic schema indexing. Features include automatic collection creation and vector schema provisioning (LuxirCollectionInitializer), OIS v1.1 tombstone deletion handling (delete_ids), zero-trust ACL security tokens, Admin UI integration, and decoupled Kafka writer consumer execution (LuxirStoreWriterConsumer).

For full configuration reference, schema details, and queries, see the Luxir Output Connector Wiki Page.

All properties are bound under spring.opencrawling.output.luxir.*:

# Spin up standalone Luxir container via Docker Compose
docker compose -f oc-luxir-output-connector/docker/docker-compose.yml up -d

# Decoupled Kafka pipeline test with Luxir
./scripts/test-luxir-decoupled.sh

Module: oc-seatunnel-output-connector
Class: SeaTunnelOutputConnector
Spring Bean: @Component (activated when spring.opencrawling.output.type=seatunnel)

Integrates Apache SeaTunnel (v2.3.13) into OpenCrawling as a distributed data integration engine and fan-out orchestrator. It enables high-throughput document crawling, text extraction, token chunking, zero-trust ACL security mapping, and dense vector embeddings to fan out directly into Apache SeaTunnel pipelines powered by the Zeta Engine, Apache Flink, or Apache Spark.

Through SeaTunnel, OpenCrawling streams can be routed to over 100+ destination sinks simultaneously (ClickHouse, Milvus, Apache Iceberg, StarRocks, Qdrant, Elasticsearch, etc.). Features automated HOCON DAG synthesis (SeaTunnelJobConfigBuilder), Zeta Engine REST API v2 orchestration (SeaTunnelRestClient), OIS v1.1 tombstone deletion handling (-D / RowKind.DELETE), stock SeaTunnel cluster compatibility without requiring custom JARs, Admin UI dynamic health checking, and decoupled Kafka writer consumer execution (SeaTunnelStoreWriterConsumer).

For full configuration reference, schema details, and queries, see the Apache SeaTunnel Output Connector Wiki Page.

All properties are bound under spring.opencrawling.output.seatunnel.*:

# Spin up standalone Apache SeaTunnel Zeta cluster via Docker Compose
docker compose -f oc-seatunnel-output-connector/docker/docker-compose.yml up -d

# Decoupled Kafka pipeline test with Apache SeaTunnel
./scripts/test-seatunnel-decoupled.sh

Module: oc-ozone-output-connector
Class: org.opencrawling.ozone.OzoneOutputConnector (decoupled writer: OzoneMigrationWriterConsumer)
Spring Bean: @Component (activated when spring.opencrawling.output.type=ozone)

Migrates enterprise content bit for bit into Apache Ozone volumes and buckets. It runs only in Migration Mode (opencrawling.pipeline.mode=migration, or pipelineMode: migration per job): Tika extraction, chunking and embeddings are skipped. A job that uses this output in RAG mode is rejected with HTTP 409.

Each UPSERT writes the original binary (SHA-256 computed while streaming), a <key>.ois.json sidecar with metadata, Zero-Trust ACLs and contentRef, and an internal index entry. DELETE removes them (DELETE_KEY) or also writes .tombstones/<key>.ois.json (ARCHIVE_TOMBSTONE). There are two transports: NATIVE (Ozone RPC, the default and about 30% faster) and S3G (S3 Gateway, with parallel multipart uploads for large objects).

For architecture, object layout, tuning and measurements, see the Apache Ozone Output Connector Wiki Page.

All properties are bound under spring.opencrawling.output.ozone.*:

Related: spring.opencrawling.kafka.topic.partitions (3) and spring.opencrawling.crawler.concurrency (4). See the Configuration Guide.

# Migration mode unit + live checks
./scripts/test-ozone-migration.sh

# Decoupled migration: source Ozone (claim check) -> Kafka -> writer -> target Ozone
./scripts/test-ozone-migration-decoupled.sh

# Same, over the S3 transports
OUTPUT_OZONE_CLIENT_TYPE=S3G CLAIM_CHECK_OZONE_CLIENT_TYPE=S3 ./scripts/test-ozone-migration-decoupled.sh

Transformation connectors are not Java classes but named configurations stored in connectors.json and managed via the Admin UI. They define which embedding engine and model the oc-embedding-service should use when processing chunks from Kafka.

Runs a local embedding model via the Ollama server. No API key required.

Recommended models by dimension:

Uses the OpenAI Embeddings API. Requires an API key.

To implement a custom Repository Connector:

1. Create a new Maven module (e.g. oc-s3-repository-connector).
2. Add oc-core as a dependency.
3. Implement RepositoryConnector:

@Component
public class S3RepositoryConnector implements RepositoryConnector {

    @Override
    public String getName() { return "S3Connector"; }

    @Override
    public void connect() throws Exception { /* initialise S3 client */ }

    @Override
    public void disconnect() throws Exception { /* shutdown */ }

    @Override
    public Flux<RepositoryDocument> scan(String basePath) {
        return Flux.create(sink -> {
            // list S3 objects under basePath prefix
            // for each object: sink.next(new RepositoryDocument(...))
            sink.complete();
        });
    }
}

4. Add the module to the root pom.xml and to oc-runtime's dependencies.
5. Register it in docker-compose-decoupled.yml with CONNECTOR_TYPE: s3.

See Repository Connector Execution Modes for how connectors are wired into the runtime.