SpiderFoot v6.0.0 implements a microservices-only architecture:

• 23 containers behind a Traefik v3 reverse proxy with full observability, AI agents, Celery task processing, and React SPA frontend
• Native async I/O via aiohttp + aiodns for high-throughput scanner modules
• AI structured outputs via Pydantic schemas and OpenAI json_schema mode
• TypeScript SDK auto-generated from the OpenAPI spec via @hey-api/openapi-ts
• PEP 561 strict typing with mypy enforcement across the Python codebase

Note: Monolith mode was removed in v6.0.0 (Batches 34–36). SpiderFoot now requires Docker Compose or Kubernetes.

┌─────────────────────────────────────────────────────────────────────────┐
│                     Traefik v3 Gateway (:443)                           │
│       Auto-TLS · Rate limiting · Reverse proxy · Path routing           │
├──────────────┬──────────────┬──────────────┬───────────────┬────────────┤
│ Frontend     │ REST API     │  Agents      │ Celery Flower │ Grafana    │
│ React SPA    │ :8001        │  :8100       │ :5555         │ :3000      │
├──────────────┴──────────────┴──────────────┴───────────────┴────────────┤
│                    Task Processing                                      │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐                          │
│  │ Celery     │ │ Celery     │ │ Apache     │                          │
│  │ Worker     │ │ Beat       │ │ Tika :9998 │                          │
│  │ (async)    │ │ (scheduler)│ │ (document) │                          │
│  └────────────┘ └────────────┘ └────────────┘                          │
├───────────────────────────────────────────────────────────────────────┤
│                       Data Layer                                        │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐          │
│  │ PostgreSQL │ │   Redis    │ │  Qdrant    │ │   MinIO    │          │
│  │ :5432      │ │  :6379     │ │ :6333      │ │ :9000/9001 │          │
│  │ (Primary)  │ │ (EventBus, │ │ (Vector    │ │ (S3 Object │          │
│  │            │ │ Cache,     │ │  Search)   │ │  Storage)  │          │
│  │            │ │ Broker)    │ │            │ │            │          │
│  └────────────┘ └────────────┘ └────────────┘ └────────────┘          │
├───────────────────────────────────────────────────────────────────────┤
│                     LLM Gateway                                        │
│  ┌────────────┐                                                        │
│  │  LiteLLM   │ Multi-provider proxy (OpenAI, Anthropic, Ollama)       │
│  │  :4000     │ Cost tracking · Model routing · Redis cache            │
│  └────────────┘                                                        │
├───────────────────────────────────────────────────────────────────────┤
│                   Observability Pipeline                                │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐          │
│  │ Vector.dev │ │   Loki     │ │ Prometheus │ │   Jaeger   │          │
│  │ :8686      │ │  :3100     │ │  :9090     │ │  :16686    │          │
│  │ :4317/4318 │ │ (Log aggr) │ │ (Metrics)  │ │ (Tracing)  │          │
│  │ :9598      │ │            │ │            │ │            │          │
│  │ (Telemetry │ └────────────┘ └────────────┘ └────────────┘          │
│  │  pipeline) │ ┌────────────┐                                         │
│  └────────────┘ │  Grafana   │ Dashboards · Alerting · Data explorer   │
│                 │  :3000     │                                          │
│                 └────────────┘                                          │
├───────────────────────────────────────────────────────────────────────┤
│                   Sidecars & Infrastructure                             │
│  ┌────────────┐ ┌────────────┐ ┌─────────────────┐                     │
│  │ pg-backup  │ │ minio-init │ │ Docker Socket   │                     │
│  │ (cron)     │ │ (one-shot) │ │ Proxy (Traefik) │                     │
│  │ → MinIO    │ │            │ │                 │                     │
│  └────────────┘ └────────────┘ └─────────────────┘                     │
└───────────────────────────────────────────────────────────────────────┘

The spiderfoot/ package is organized into 23 domain sub-packages plus several top-level modules:

Top-level modules: celery_app.py, helpers.py, result_cache.py, retry.py, service_integration.py, service_registry.py, service_runner.py, target.py, threadpool.py, workspace.py

# Preferred: import from subpackage init
from spiderfoot.events import SpiderFootEvent
from spiderfoot.plugins import SpiderFootPlugin
from spiderfoot.config import SF_DATA_TYPES

# Also valid: import from specific module
from spiderfoot.events.event import SpiderFootEvent
from spiderfoot.scan.scan_state import SpiderFootScanState

# Top-level re-exports still work
from spiderfoot import SpiderFootEvent, SpiderFootPlugin

Note (v5.245.0): All backward-compatibility shim files in the spiderfoot/ root were removed. Code that used old paths like from spiderfoot.event import ... or from spiderfoot.plugin import ... must update to the subpackage paths shown above.

┌─────────────────────────────────────────────────────────────────────┐
│                    Traefik v3 Gateway (:443)                        │
│            Auto-TLS · Rate limiting · Reverse proxy                │
├──────────┬──────────────┬──────────────┬──────────────┬────────────┤
│ Frontend │ REST API     │  Agents      │ Celery       │ Tika       │
│ React SPA│ :8001        │  :8100       │ Workers/Beat │ :9998      │
├──────────┴──────────────┴──────────────┴──────────────┴────────────┤
│                      Service Layer                                 │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐      │
│  │ServiceReg. │ │ConfigSvc   │ │ Metrics    │ │ Structured │      │
│  │(DI Contain)│ │(env+file)  │ │(Prometheus)│ │  Logging   │      │
│  └────────────┘ └────────────┘ └────────────┘ └────────────┘      │
├───────────────────────────────────────────────────────────────────┤
│                     Core Services                                  │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐      │
│  │HttpService │ │ DnsService │ │CacheService│ │DataService │      │
│  │(pooled HTTP│ │(DNS+cache) │ │(Mem/File/  │ │(PostgreSQL/│      │
│  │ +proxy)    │ │            │ │  Redis)    │ │  gRPC)     │      │
│  └────────────┘ └────────────┘ └────────────┘ └────────────┘      │
├───────────────────────────────────────────────────────────────────┤
│                   Execution Layer                                  │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐      │
│  │ WorkerPool │ │   Scan     │ │Correlation │ │ EventBus   │      │
│  │(Thread/Proc│ │ Scheduler  │ │  Service   │ │(Mem/Redis/ │      │
│  │  pool)     │ │(priority Q)│ │(auto+batch)│ │  NATS)     │      │
│  └────────────┘ └────────────┘ └────────────┘ └────────────┘      │
├───────────────────────────────────────────────────────────────────┤
│                    LLM Gateway                                     │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐                      │
│  │  LiteLLM   │ │  AI Agents │ │ OTel       │                      │
│  │(multi-LLM  │ │ (6 agents) │ │ Tracing    │                      │
│  │  proxy)    │ │            │ │ (Vector→   │                      │
│  └────────────┘ └────────────┘ │  Jaeger)   │                      │
│                                 └────────────┘                      │
├───────────────────────────────────────────────────────────────────┤
│                    Data Pipeline                                   │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐      │
│  │Vector.dev  │ │   Loki     │ │ Prometheus │ │  Grafana   │      │
│  │(logs/traces│ │ (log aggr) │ │ (metrics)  │ │(dashboards)│      │
│  │ /metrics)  │ │            │ │            │ │            │      │
│  └────────────┘ └────────────┘ └────────────┘ └────────────┘      │
├───────────────────────────────────────────────────────────────────┤
│                     Module Layer                                   │
│  ┌──────────────────────┐ ┌──────────────────────┐                 │
│  │ SpiderFootPlugin     │ │SpiderFootModernPlugin│                 │
│  │ (legacy, 309         │ │(service-aware, new   │                 │
│  │  modules)            │ │ modules)             │                 │
│  └──────────────────────┘ └──────────────────────┘                 │
└───────────────────────────────────────────────────────────────────┘

Abstracted publish/subscribe messaging with three backends:

• Memory (default): In-process asyncio.Queue-based, zero config
• Redis Streams: Consumer groups, at-least-once delivery
• NATS JetStream: High-throughput, durable messaging

Topics follow a dot-notation pattern: scan.started, scan.completed, module.event.new, etc.

Database abstraction layer supporting:

• Local: Direct PostgreSQL via SpiderFootDb
• HTTP: Remote REST API for microservices mode
• gRPC: High-performance binary protocol

Provides CRUD for scans, events, configs, and correlations.

Extracted HTTP client from the legacy SpiderFoot god object:

• Connection pooling
• SOCKS4/5 and HTTP proxy support
• Configurable timeouts and user-agents
• TLS certificate parsing
• Google/Bing API pagination helpers

Extracted DNS resolver with built-in TTL cache:

• A/AAAA/MX/TXT/NS/PTR record types
• Wildcard detection
• Zone transfer checks
• Configurable resolvers

Three-tier caching:

• Memory: LRU eviction, TTL-based expiry
• File: SHA-224 hashed filenames, persistent
• Redis: Distributed cache for microservices

Centralized configuration management:

• 40+ environment variable mappings
• Type coercion (bool, int, float, str)
• Validation rules (min/max, choices, required)
• Hot-reload with watcher callbacks
• Snapshot isolation for per-scan configs

Dependency injection container:

• Lazy factory pattern (services created on first access)
• Well-known service constants
• Thread-safe singleton
• ServiceMixin for convenient property access

Module execution infrastructure:

• Thread pool (default) or process pool strategies
• Per-module workers with event queues
• Health monitoring with automatic restart
• Graceful shutdown with drain

Scan lifecycle management:

• Priority queue (CRITICAL > HIGH > NORMAL > LOW)
• Concurrent scan limiting
• Pause/resume/abort controls
• Timeout detection
• Progress tracking

Standalone correlation engine:

• Wraps the existing RuleExecutor pipeline
• EventBus subscription for auto-trigger on scan completion
• Async queue for batch processing
• Result caching and callbacks
• Prometheus metrics integration

Unified request routing:

• Circuit breaker per downstream service
• Token-bucket rate limiting per client
• Microservices-only mode (monolith removed in v6.0.0)
• FastAPI router integration
• System status aggregation

Zero-dependency Prometheus-compatible instrumentation:

• Counter, Gauge, Histogram types
• Label support for dimensional metrics
• /metrics endpoint in Prometheus text format
• Pre-defined metrics for scans, events, HTTP, DNS, cache

Data pipeline to external systems:

• Batched HTTP posting to Vector.dev
• Separate channels for events, logs, metrics
• Configurable transforms and routing
• Elasticsearch/S3/file sinks via Vector config

Inter-service communication:

• Protobuf service contracts (proto/spiderfoot.proto)
• JSON-over-HTTP fallback when gRPC unavailable
• ServiceDirectory for environment-based endpoint discovery
• Health checking

The docker-compose.yml defines 23 containers across 8 profiles:

• sf-frontend: Traefik ↔ Frontend/API (external-facing)
• sf-backend: All services ↔ PostgreSQL/Redis/Qdrant/MinIO (internal)

The web interface is a modern React SPA built with TypeScript, Vite, and Tailwind CSS. It features a dark theme with cyan accents, responsive layout, and real-time scan updates via GraphQL subscriptions. All HTML injection points are sanitized with DOMPurify, and ESLint with typescript-eslint enforces strict type safety (zero any types).

Key pages and features:

Scan Detail component architecture: The Scan Detail page is built from 10 focused tab components in frontend/src/components/scan/, each owning its own state, queries, and rendering. The parent ScanDetail.tsx is a ~130-line shell that handles routing and tab navigation. Shared utilities live in frontend/src/lib/ (geo constants, HTML sanitization, error handling).

Six LLM-powered analysis agents that subscribe to Redis event bus events and produce structured intelligence. All agents extend BaseAgent with concurrency control, timeout handling, and Prometheus metrics.

All agents route LLM calls through LiteLLM (:4000) for unified model selection, cost tracking, and provider failover.

Converts documents to text, extracts entities and IOCs, and stores results in MinIO.

PDF (pypdf), DOCX (python-docx), XLSX (openpyxl), HTML, RTF (striprtf), plain text. Optional Apache Tika fallback for complex documents.

Pre-compiled regex patterns for: IPv4/IPv6, emails, URLs, domains, MD5/SHA1/SHA256 hashes, phone numbers, CVEs, Bitcoin/Ethereum addresses, AWS keys, credit cards. Smart deduplication and private IP filtering.

Allows users to supply their own documents, IOCs, reports, and context data to augment automated OSINT collection.

Unified proxy supporting 100+ LLM providers through an OpenAI-compatible API. Configuration in infra/litellm/config.yaml.

• default → gpt-4o-mini
• fast → gpt-3.5-turbo
• smart → gpt-4o
• local → ollama/llama3

Vector.dev replaces both Promtail and OpenTelemetry Collector as a unified telemetry pipeline:

• Logs: Docker container logs → JSON parse → route by level → Loki + MinIO
• Events: HTTP source (:8686) → enrich with category/risk → MinIO + file
• Metrics: Internal metrics → Prometheus exporter (:9598)
• Traces: OTLP receiver (:4317 gRPC, :4318 HTTP) → forward to Jaeger

Five pre-provisioned dashboards in infra/grafana/dashboards/:

spiderfoot-api, spiderfoot-scanner, spiderfoot-agents, spiderfoot-enrichment, vector, qdrant, minio, jaeger, litellm, prometheus (self-monitoring).

OpenTelemetry instrumentation via spiderfoot/observability/tracing.py with graceful no-op fallback when SDK not installed. Traces flow: Service → OTLP → Vector.dev :4317 → Jaeger :4317.

Custom HTTP-based Qdrant client (NOT the PyPI qdrant-client). Communicates with Qdrant via urllib.request REST calls.

• Singleton via get_qdrant_client() / init_qdrant_client()
• Backends: MemoryVectorBackend (testing), HttpVectorBackend (production)
• Collection prefix: sf_ (configurable via SF_QDRANT_PREFIX)
• Key classes: VectorPoint(id, vector, payload, score), SearchResult(points, query_time_ms, total_found), Filter(must, must_not, should) with match() / range() statics, CollectionInfo(name, vector_size, distance, point_count)
• Methods: ensure_collection, search, upsert, get, delete, scroll, count, collection_info, list_collections

Generates vector embeddings for text data:

• Providers: MOCK (default), SENTENCE_TRANSFORMER, OPENAI, HUGGINGFACE
• Default model: all-MiniLM-L6-v2 (384 dimensions)
• Methods: embed_text(), embed_texts() with caching and batching

5 correlation strategies over vectorized scan events:

Default collection: sf_osint_events

S3-compatible object storage for artifacts, reports, and backups.

• MinioStorageClient: Upload, download, list, delete, presigned URLs
• Singleton: get_minio_client() with automatic bucket creation
• Lifecycle: Configurable retention policies per bucket

Runs in the sf-pg-backup container:

• Hourly pg_dump of the SpiderFoot database
• Compressed archives uploaded to the spiderfoot-backups bucket
• Configurable retention (default: 7 days)
• Health check via backup recency validation

Code-first GraphQL using Strawberry ≥ 0.235.0, mounted at /api/graphql with GraphiQL IDE.

Protocols: graphql-transport-ws, graphql-ws

• QueryDepthLimiter: Max depth = 10 (prevents deeply nested abuse)
• DataLoaders: ScanEventLoader, ScanCorrelationLoader (N+1 prevention)

All 309 existing modules continue to work unchanged. They use self.sf (the SpiderFoot god object) for HTTP, DNS, and other operations.

New modules can extend SpiderFootModernPlugin to access services directly:

from spiderfoot.plugins.modern_plugin import SpiderFootModernPlugin

class sfp_example(SpiderFootModernPlugin):
    def handleEvent(self, event):
        # Modern: uses HttpService with metrics
        res = self.fetch_url("https://api.example.com/lookup")

        # Cache results
        self.cache_put("key", data, ttl=3600)

        # DNS resolution via DnsService
        addrs = self.resolve_host(hostname)

See MODULE_MIGRATION_GUIDE.md for step-by-step migration instructions.

JWT, API key, and Basic authentication with role-based access control (ADMIN, ANALYST, VIEWER, API roles). Pluggable into any ASGI/WSGI app.

Multi-format scan result export: JSON, CSV, STIX 2.1 bundles, and SARIF for integration with CI/CD security tooling.

Real-time scan event streaming over WebSocket with channel-based subscriptions per scan, module, or event type.

Multi-channel alerting with wildcard topic subscriptions, supporting Slack webhooks, generic webhooks, SMTP email, and log output.

Secure API key and credential storage with four backends: in-memory, environment variables, plain JSON file, and encrypted file (PBKDF2 + XOR). Includes rotation tracking, access auditing, and config injection.

Centralised error capture with fingerprint-based deduplication, automatic classification (network/auth/parse/timeout/etc.), sliding-window rate tracking, and configurable alert thresholds with callbacks.

Bounded priority queue (HIGH/NORMAL/LOW) with backpressure support. Three overflow strategies (BLOCK/REJECT/DROP_OLDEST), batch dequeue, retry tracking with dead-letter queue.

Runtime dependency resolution for modules. Given desired output event types, walks backwards through the event dependency chain to compute the minimal module set and topological load order.

Version-controlled schema evolution with numbered migration files, upgrade/downgrade functions, dry-run mode, and checksum validation. Supports PostgreSQL dialect.

Registry-driven module loading adapter that replaces the scanner’s legacy __import__ loop with ModuleRegistry for discovery/instantiation and ModuleGraph for topological dependency ordering. Features:

• Registry-first loading with automatic legacy fallback
• Topological execution order via Kahn’s algorithm (replaces _priority sort)
• Minimal-set pruning — when desired output types are specified, only modules in the dependency chain are loaded
• Cycle detection with warnings (cycles don’t break execution)
• LoadResult dataclass with detailed statistics: loaded/failed/skipped counts, ordering method, pruning info, and timing
• Global singleton with thread-safe init_module_loader() / get_module_loader() / reset_module_loader()
• Wired into scanner via service_integration._wire_module_loader()

Clean abstraction over SpiderFootDb using the Repository Pattern. Replaces 20+ direct SpiderFootDb(config) instantiations across API routers with injectable, testable repository instances.

• AbstractRepository — Base class with context-manager lifecycle, dbh property, is_connected, close(), __enter__/__exit__
• ScanRepository — Scan CRUD: create_scan(), get_scan(), list_scans(), update_status(), delete_scan(), config, logs, errors. Includes ScanRecord dataclass with from_row()/to_dict()
• EventRepository — Event/result operations: store_event(), get_results(), get_unique_results(), get_result_summary(), search(), element sources/children (direct + recursive), false-positive management, batch log events
• ConfigRepository — Global config: set_config(), get_config(), clear_config()
• RepositoryFactory — Creates repos with shared or per-request DB handles. Thread-safe singleton via init_repository_factory() / get_repository_factory() / reset_repository_factory()
• FastAPI Depends providers — get_scan_repository(), get_event_repository(), get_config_repository() in api/dependencies.py with automatic lifecycle management
• Wired into scanner via service_integration._wire_repository_factory()

Starlette/FastAPI middleware bridging the existing RateLimiterService into the API layer. Every incoming request is checked against per-tier rate limits before reaching the router.

• Per-client identity extraction from API key, X-Forwarded-For, or direct IP
• Route-tier mapping — /api/scans → scan tier, /api/data → data tier, etc.
• 429 Too Many Requests with Retry-After header on limit exceeded
• X-RateLimit-* response headers (Limit, Remaining, Reset) on every response
• Exempt paths for health checks, docs, OpenAPI spec
• Per-client buckets — different API keys/IPs have independent limits
• RateLimitStats with tier-level and top-offender tracking
• install_rate_limiting(app, config) wiring function
• Installed in api/main.py after request tracing middleware

Standardized pagination across all API list endpoints with consistent request parameters and response envelopes.

• PaginationParams — FastAPI Depends()-compatible query extractor supporting page-based (page/page_size) and offset-based (offset/limit) modes with automatic mapping between them
• PaginatedResponse — Standardized envelope: items, total, page, page_size, pages, has_next, has_previous
• paginate() — In-memory slicing with optional sort support
• paginate_query() — For pre-sliced DB results with total count
• Sort helpers — dict_sort_key(), attr_sort_key() for common patterns
• RFC 8288 Link headers — generate_link_header() for next/prev/ first/last navigation
• make_params() — Convenience constructor for programmatic/test use

Rewritten to delegate to CorrelationService instead of raw SpiderFootDb / config manipulation. All 7 endpoints now use the service layer and Cycle 25 pagination.

• Rule CRUD — add_rule(), get_rule(), update_rule(), delete_rule(), filter_rules() added to CorrelationService
• get_correlation_svc — FastAPI Depends() provider in dependencies.py returning the singleton service
• Real execution — Test/analyze endpoints call svc.run_for_scan() with actual timing instead of hardcoded results
• Pagination — List and detailed endpoints use PaginationParams
  • paginate() for standardized response envelopes
• No direct DB access — All SpiderFootDb(config.get_config()) and json.dumps()/configSet() calls eliminated from the router

Unified scan lifecycle management combining ScanRepository (Cycle 23) with ScanStateMachine for formal state-transition enforcement.

• ScanService — High-level facade wrapping repository + state machine with CRUD methods: list_scans, get_scan, create_scan, delete_scan, delete_scan_full, stop_scan, get_scan_state
• State Machine Integration — stop_scan() validates transitions (RUNNING→STOPPING, CREATED→CANCELLED) before persisting; returns HTTP 409 Conflict when transition is invalid (e.g. stopping a completed scan)
• get_scan_service — FastAPI Depends() generator provider in dependencies.py with automatic lifecycle management
• Pagination — list_scans endpoint uses PaginationParams + paginate() for standardized response envelopes
• Typed records — Endpoints return ScanRecord.to_dict() instead of raw tuple-index dicts (scan[0], scan[6], etc.)
• Gradual migration — Service exposes .dbh for endpoints not yet migrated (export, viz); full migration planned for future cycles

Centralises all SpiderFootDb instantiation across the CherryPy WebUI into a single overridable _get_dbh() method.

• 78 per-request SpiderFootDb(self.config) calls replaced across scan.py (62), export.py (7), settings.py (4), helpers.py (1), info.py (1), routes.py (3)
• DbProvider mixin added to WebUiRoutes MRO — all endpoint classes inherit _get_dbh(config=None) via diamond inheritance
• Single override point — tests or future service migration can replace _get_dbh() instead of patching 78+ instantiation sites
• Config override — _get_dbh(cfg) for cases needing deepcopy config (e.g. rerunscan, rerunscanmulti)
• CHANGELOG — event_schema.py entry annotated as deleted in v5.33.0

Completes the ScanService facade migration started in Cycle 27, eliminating all raw SpiderFootDb instantiation from the scan router.

• 15 new ScanService methods — get_events(), search_events(), get_correlations(), get_scan_logs(), get_metadata()/set_metadata(), get_notes()/set_notes(), archive()/unarchive(), clear_results(), set_false_positive(), get_scan_options()
• All 25 scan endpoints now delegate to ScanService via Depends(get_scan_service) — zero SpiderFootDb imports remain
• Export endpoints — CSV/XLSX event export, multi-scan JSON export, search export, logs export, correlations export
• Lifecycle endpoints — create, rerun, rerun-multi, clone
• Results management — false-positive with parent/child validation, clear results
• Metadata/notes/archive — CRUD with hasattr guards for optional DB methods
• Static route ordering — /scans/export-multi, /scans/viz-multi, /scans/rerun-multi registered before /{scan_id} routes

Removes the last 3 raw SpiderFootDb instantiations from all API routers, achieving a clean architectural boundary between the router layer and the database.

• config.py — GET /event-types now uses ConfigRepository via Depends(get_config_repository). New ConfigRepository.get_event_types() method wraps dbh.eventTypes().
• reports.py — _get_scan_events() helper rewritten to accept an injected ScanService. Endpoints POST /reports/generate and POST /reports/preview pass their Depends(get_scan_service) instance.
• websocket.py — _polling_mode() rewritten to build a ScanService from RepositoryFactory instead of raw SpiderFootDb. Proper cleanup via svc.close() in finally block.
• Dead code removal — event_schema.py (655 lines) and its test file deleted (zero production imports).
• Verification — grep -r "SpiderFootDb" spiderfoot/api/routers/ returns only docstring/comment references, zero actual imports.

Dedicated service layer for scan data visualization, removing raw SpiderFootDb from all 5 visualization router endpoints.

• VisualizationService — Composes ScanRepository + raw dbh with methods: get_graph_data, get_multi_scan_graph_data, get_summary_data, get_timeline_data, get_heatmap_data
• Smart scan validation — _require_scan() uses ScanRepository first, with raw dbh.scanInstanceGet() fallback
• Timeline aggregation — Handles both datetime and epoch timestamps; supports hour/day/week bucketing
• Heatmap matrix — Builds x/y matrix from result dimensions (module/type/risk) with configurable axes
• Multi-scan graph — Merges results across scan IDs, skipping invalid scans with warning logs
• get_visualization_service — FastAPI Depends() generator provider in dependencies.py with automatic lifecycle management
• Static route ordering — /visualization/graph/multi registered before /{scan_id} to avoid path parameter capture

Central fan-out hub bridging the EventBus to WebSocket/SSE consumers. Per-scan consumer queues with bounded overflow (drop-oldest policy), EventBus subscription management, and lifecycle helpers for scan_started / scan_completed / status_update events.

Lightweight synchronous adapter that sits in the scanner's waitForThreads() dispatch loop. Forwards each SpiderFootEvent to the EventRelay for real-time WebSocket delivery. Features: configurable per-event-type throttling, large-data truncation, per-scan statistics, and a bridge registry for lifecycle management.

Starlette middleware that generates/echoes X-Request-ID headers, sets contextvars for request context, and logs request start/end with timing. Warns on slow requests exceeding a configurable threshold.

Outbound HTTP notification delivery with HMAC-SHA256 signing (X-SpiderFoot-Signature), exponential backoff retries, delivery history (bounded deque), and stats. Uses httpx with urllib fallback.

Webhook CRUD operations, event routing to matching/enabled webhooks, fire-and-forget async delivery, webhook testing, and integration with the Task Queue and Alert Engine for automated notifications.

ThreadPoolExecutor-backed task execution with TaskRecord state machine (PENDING → RUNNING → COMPLETED/FAILED/CANCELLED), progress tracking, completion callbacks, and a singleton task manager.

11-section typed dataclass configuration replacing the legacy flat dict. Sections: Core, Network, Database, Web, API, Cache, EventBus, Vector, Worker, Redis, Elasticsearch. Features: from_dict() / to_dict() round-trip, apply_env_overrides() for SF_* variables, 20+ validation rules, and merge semantics for layered overrides.

High-performance HTTP/DNS client layer built on aiohttp + aiodns:

• Session pool: aiohttp.ClientSession instances cached per (module_name, event_loop) tuple — avoids recreating connections across calls
• async_fetch_url(): Returns dict matching the sync fetchUrl() shape (code, status, content, headers, realurl)
• DNS resolver: aiodns.DNSResolver for A/AAAA/PTR lookups with getaddrinfo() fallback
• Wildcard detection: async_check_dns_wildcard() for brute-force flood mitigation
• Plugin integration: SpiderFootAsyncPlugin subclasses call async_fetch_url() / async_resolve_host() directly — no run_in_executor wrapping

See Async Plugin Guide for module development.

Pydantic-validated structured output pipeline for LLM responses:

• 12 response models: ScanReportOutput, ExecutiveSummaryOutput, RiskAssessmentOutput, ThreatAssessmentOutput, CorrelationOutput, FindingValidationOutput, TextSummaryOutput, plus nested types (Finding, ThreatIndicator, Recommendation, Attribution)
• chat_structured() in LLMClient: Builds OpenAI response_format: json_schema payload from model.model_json_schema(), injects format hint into system message, validates response via model_validate()
• call_llm_structured() in AgentBase: Same pipeline available to AI agents
• Mock support: _MockGenerator._mock_from_schema() produces schema-conformant test data

See AI Structured Outputs Guide for usage.

Auto-generated, type-safe API client for the React frontend:

• Generator: @hey-api/openapi-ts v0.92.4 with native fetch client (not Axios)
• Post-processing: dump_openapi.py normalizes operationId values for clean SDK method names
• JWT interceptor: client.interceptors.request.use() attaches Authorization: Bearer <token> to every request
• Regeneration: npm run generate:api re-generates from the live /api/openapi.json

See TypeScript SDK Guide for the generation workflow.

Pipeline-friendly scan result export:

• Endpoint: GET /api/scans/{id}/export/jsonl — streams newline-delimited JSON
• SSE stream: GET /events/stream — Server-Sent Events for real-time scan event delivery
• Event enrichment: SpiderFootEvent.asDict() produces full serializable event dicts
• Use cases: jq pipelines, Elasticsearch bulk ingest, SIEM integration, real-time dashboards

See Streaming Export Guide for integration examples.

Static type safety enforcement across the Python codebase:

• py.typed marker in spiderfoot/ — enables downstream type checking for packages that import SpiderFoot
• mypy config in setup.cfg: python_version=3.11, check_untyped_defs=true, no_implicit_optional=true
• Coverage: DB layer (14 return types + 17 params fixed), core models (__all__ exports), network utilities (implicit Optional → explicit union types)
• Convention: Use X | None (PEP 604) instead of Optional[X] for all new code