Author: poppopjmp

SpiderFoot can be configured via the web UI, configuration file, and environment variables. Proper configuration ensures you get the most out of all 309 modules and integrations, including TikTok OSINT, blockchain analytics, AI analysis, and performance optimization features, and helps tailor SpiderFoot to your environment and use case.

Many modules require API keys for external services. Configure these in the web UI:

• Go to Settings → Module Settings
• Enter your API keys for services such as:
  • VirusTotal
  • Shodan
  • Hunter.io
  • SecurityTrails
  • Have I Been Pwned
  • TikTok Research API (for enhanced TikTok OSINT)
  • BlockCypher API (for blockchain analytics)
  • Etherscan API (for Ethereum analysis)
  • OpenAI API (for AI threat intelligence summaries)
  • ...and more

Tip: Modules that require API keys will show a warning if not configured. You can run scans without API keys, but results may be limited. The enhanced modules provide fallback web scraping when API keys are not available.

Advanced options can be set in spiderfoot.conf (or spiderfoot.cfg):

[webui]
host = 127.0.0.1
port = 5001

[database]
path = spiderfoot.db

[security]
# CSRF Protection
csrf_enabled = True
csrf_secret_key = your-strong-secret-key
csrf_timeout = 3600

# Rate Limiting
rate_limiting_enabled = True
rate_limiting_storage = memory
rate_limiting_api_requests_per_minute = 60
rate_limiting_web_requests_per_minute = 120

# Input Validation
input_validation_enabled = True
input_validation_max_input_length = 10000
input_validation_strict_mode = False

# Session Security
session_security_enabled = True
session_security_storage = memory
session_security_session_timeout = 3600
session_security_max_sessions_per_user = 5

# API Security
api_security_enabled = True
api_security_jwt_secret = your-jwt-secret-key
api_security_jwt_expiry = 3600
api_security_api_key_length = 32

# Security Logging
security_logging_enabled = True
security_logging_log_file = security.log
security_logging_log_level = INFO

# Security Headers
security_headers_enabled = True

• You can change the web UI port, database location, logging options, security features, and more.
• For production, consider moving the database to a persistent storage location.
• Security: Always use strong, unique secret keys for CSRF and JWT tokens.

You can override some settings using environment variables (useful for Docker and CI/CD):

• SPIDERFOOT_DB_PATH – Path to the database file
• SPIDERFOOT_WEBUI_PORT – Port for the web UI
• SPIDERFOOT_WEBUI_HOST – Host address for the web UI

• SPIDERFOOT_CSRF_SECRET – CSRF protection secret key
• SPIDERFOOT_JWT_SECRET – JWT token secret key
• SPIDERFOOT_RATE_LIMIT_STORAGE – Rate limiting storage backend (memory/redis)
• SPIDERFOOT_REDIS_HOST – Redis server host (if using Redis storage)
• SPIDERFOOT_REDIS_PORT – Redis server port
• SPIDERFOOT_SECURITY_LOG_FILE – Path to security log file
• SPIDERFOOT_SESSION_TIMEOUT – Session timeout in seconds

• VIRUSTOTAL_API_KEY – VirusTotal API key
• SHODAN_API_KEY – Shodan API key
• HUNTER_API_KEY – Hunter.io API key
• SECURITYTRAILS_API_KEY – SecurityTrails API key
• HIBP_API_KEY – Have I Been Pwned API key
• SPIDERFOOT_LOG_LEVEL – Logging verbosity (e.g., INFO, DEBUG)

• Always keep your API keys secure and never share them publicly.
• Use a dedicated config file for production deployments.
• Regularly review and update your API keys and module settings.
• For Docker, use environment variables or mount a config file for persistent configuration.

• If a module fails, check if its API key is set and valid.
• For config file errors, ensure correct INI syntax and file permissions.
• For Docker, use environment variables or mount a config file.
• See the Troubleshooting Guide for more help.

SpiderFoot uses PostgreSQL as its database backend.

• PostgreSQL is required for all deployments (monolith and microservices).

• Set the database connection string in your config file or environment variable (e.g., SPIDERFOOT_DB_TYPE=postgresql and SPIDERFOOT_DB_PATH=postgresql://user:pass@host/dbname).

• Schema creation and migrations are idempotent. You can safely run SpiderFoot with an existing PostgreSQL schema.
• All upsert/replace operations use PostgreSQL-native ON CONFLICT for correct, atomic updates.
• Composite keys and unique constraints are enforced where required for upsert support. For example, tbl_scan_config uses a unique constraint on (scan_instance_id, component, opt).
• Schema versioning is automatic. The schema version is tracked in the tbl_schema_version table, and migrations are applied as needed.

• All database operations use granular exception handling and retry logic for transient errors (e.g., connection drops, deadlocks).
• Errors are logged with context, including the query and parameters.
• If a schema or migration error occurs, SpiderFoot will log the error and abort startup to prevent data corruption.

• Connection pooling is recommended for high concurrency (e.g., using psycopg2.pool).
• Always back up your database before upgrading SpiderFoot.
• For Docker deployments, use a managed PostgreSQL service for persistence.

• If you see errors about missing tables or constraints, ensure your database is not corrupted and that SpiderFoot has permission to create/modify tables.
• Check that your PostgreSQL user has the necessary privileges (CREATE, INSERT, UPDATE, etc.).
• Use the drop_all_tables and dump_schema helpers (see developer guide) for test isolation and debugging.

When deploying SpiderFoot in production, follow these security configuration guidelines:

1. Strong Secret Keys

# Generate cryptographically secure keys
SPIDERFOOT_CSRF_SECRET=$(openssl rand -hex 32)
SPIDERFOOT_JWT_SECRET=$(openssl rand -hex 32)

2. Rate Limiting Configuration

[security]
rate_limiting_enabled = True
rate_limiting_storage = redis  # Use Redis for distributed setups
rate_limiting_api_requests_per_minute = 60
rate_limiting_web_requests_per_minute = 120

3. Session Security

[security]
session_security_enabled = True
session_security_session_timeout = 1800  # 30 minutes
session_security_max_sessions_per_user = 3

4. Input Validation

[security]
input_validation_enabled = True
input_validation_strict_mode = True  # Enable for production
input_validation_max_input_length = 5000

Enable comprehensive security logging:

[security]
security_logging_enabled = True
security_logging_log_file = /var/log/spiderfoot/security.log
security_logging_log_level = INFO

Monitor these security events:

• Authentication failures
• Rate limit violations
• Input validation failures
• Session anomalies
• API abuse attempts

Development Environment:

export SPIDERFOOT_CSRF_SECRET=dev-csrf-secret
export SPIDERFOOT_JWT_SECRET=dev-jwt-secret
export SPIDERFOOT_SECURITY_LOG_LEVEL=DEBUG

Production Environment:

export SPIDERFOOT_CSRF_SECRET=your-production-csrf-secret
export SPIDERFOOT_JWT_SECRET=your-production-jwt-secret
export SPIDERFOOT_RATE_LIMIT_STORAGE=redis
export SPIDERFOOT_REDIS_HOST=redis.production.local
export SPIDERFOOT_SECURITY_LOG_FILE=/var/log/spiderfoot/security.log

Validate your security configuration:

cd spiderfoot
python security_validator.py /path/to/spiderfoot

This will check:

• Security module availability
• Configuration validity
• Security feature functionality
• Performance impact assessment

SpiderFoot v6.0.0 includes configuration options for enhanced modules:

[performance_optimizer]
# Cache settings
cache_enabled = True
cache_ttl_seconds = 3600
max_cache_size = 50000
cache_cleanup_interval = 300

# Rate limiting
rate_limiting_enabled = True
default_delay_seconds = 1.0
adaptive_backoff = True
max_delay_seconds = 60.0

# Resource monitoring
resource_monitoring_enabled = True
memory_threshold_mb = 1024
gc_threshold_percentage = 80.0

[advanced_correlation]
# Correlation engine settings
correlation_enabled = True
confidence_threshold = 0.7
temporal_window_hours = 24
max_entities = 10000

# Geospatial clustering
geospatial_enabled = True
clustering_distance_km = 10.0
min_cluster_size = 3

# Entity resolution
entity_resolution_enabled = True
similarity_threshold = 0.8
cross_platform_correlation = True

[blockchain_analytics]
# API configuration
blockcypher_api_key = your_blockcypher_key
etherscan_api_key = your_etherscan_key
bitcoin_api_provider = blockcypher
ethereum_api_provider = etherscan

# Analysis settings
transaction_depth = 3
risk_threshold = 0.6
sanctions_check_enabled = True
wallet_clustering_enabled = True

# Rate limiting
api_rate_limit_per_second = 3
max_concurrent_requests = 5

[tiktok_osint]
# Data collection settings
respect_robots_txt = True
user_agent_rotation = True
request_delay_seconds = 2.0
max_retries = 3

# Content analysis
profile_analysis_enabled = True
hashtag_extraction_enabled = True
verification_check_enabled = True
follower_analysis_enabled = False

# Rate limiting
requests_per_minute = 30
burst_allowance = 5

[ai_summary]
# OpenAI configuration
openai_api_key = your_openai_key
model = gpt-3.5-turbo
max_tokens = 1000
temperature = 0.3

# Summary settings
summary_frequency = on_finish
max_events_per_summary = 100
include_confidence_scores = True
threat_level_analysis = True

See the User Guide for more usage details and advanced configuration options.