This guide covers deploying SpiderFoot using Docker Compose with profile-based service activation. Start with just the 5 core services, then enable additional profiles as needed.

git clone https://github.com/poppopjmp/spiderfoot.git
cd spiderfoot

# Copy and configure environment
cp .env.example .env
# Edit .env — change passwords, uncomment profile sections as needed

# Core only (5 services)
docker compose -f docker-compose.yml up --build -d

# Full stack (all services except SSO)
docker compose -f docker-compose.yml --profile full up --build -d

# View logs
docker compose -f docker-compose.yml logs -f

# Stop everything
docker compose -f docker-compose.yml down

Core (no profile) — http://localhost:3000:

Full stack (--profile full) — https://localhost via Traefik:

Services are organized into profiles — activate only what you need:

# Mix and match profiles
docker compose -f docker-compose.yml --profile proxy --profile storage up -d

# Full stack + SSO
docker compose -f docker-compose.yml --profile full --profile sso up -d

The stack uses two Docker networks (sf-frontend, sf-backend) and organizes services by profile:

• sf-frontend — Browser-facing (Traefik, Frontend, API)
• sf-backend — Internal only (PostgreSQL, Redis, Qdrant, MinIO)

Seven buckets are auto-created by sf-minio-init on first boot:

Access the MinIO Console at https://localhost/minio/ (requires proxy + storage profiles; default credentials: minioadmin / minioadmin).

Qdrant runs on port 6333 and stores embeddings for semantic OSINT event search. Collections are prefixed with sf_.

The GraphQL API exposes semanticSearch and vectorCollections queries for searching and inspecting vector data.

Copy .env.example to .env and customise:

cp .env.example .env
# Uncomment profile-specific sections when activating profiles

Key variables:

TLS requires the proxy profile (Traefik). To enable:

# Generate a self-signed cert (testing)
./generate-certificate

# Start with proxy profile
docker compose -f docker-compose.yml --profile proxy up -d

# Traefik auto-discovers TLS certificates from the mounted certs directory

All services include Docker HEALTHCHECK directives. Monitor with:

# Check all service health
docker compose -f docker-compose.yml ps

# Detailed health for one service
docker inspect --format='{{json .State.Health}}' sf-api

API health endpoint: GET /api/health

The sf-pg-backup sidecar runs pg_dump on a cron schedule and uploads backups to the sf-pg-backups MinIO bucket.

# List backups
curl http://localhost/api/storage/buckets/sf-pg-backups

# Manual backup trigger
docker exec sf-pg-backup /scripts/pg_backup_minio.sh

For horizontal scaling, increase replicas of stateless services:

docker compose -f docker-compose.yml up -d --scale sf-api=3

The active scan worker requires the scan profile. Scale independently:

docker compose -f docker-compose.yml --profile scan up -d --scale celery-worker-active=3

Each instance competes for tasks from the scan queue via Celery's fair scheduling. See Active Scan Worker Guide for full details on the 33+ tools, resource requirements, and security considerations.

For production deployments, consider the Helm chart in helm/ for Kubernetes.

# All services
docker compose -f docker-compose.yml logs -f

# Specific service
docker logs -f sf-api

# Archived logs in MinIO
# Access via MinIO Console at https://localhost/minio/