Troubleshooting Guide - paritytech/dotlake-community GitHub Wiki

This guide helps you diagnose and resolve common issues with DotLake Community. If you don't find your issue here, check our GitHub Issues or create a new one.

🔍 Quick Diagnosis

Start here to identify your issue:

# 1. Check if all services are running
docker ps

# 2. Check service logs
docker logs dotlake-sidecar
docker logs dotlake-ingest
docker logs dotlake-postgres
docker logs dotlake-frontend

# 3. Test configuration
python3 test_config.py

# 4. Check network connectivity
curl http://localhost:8080/blocks/head

🚨 Common Issues

1. Docker Permission Errors

Symptoms:

Got permission denied while trying to connect to the Docker daemon socket

Solutions:

Option A: Add user to docker group (Recommended)

# Add user to docker group
sudo usermod -aG docker $USER

# Apply changes (log out and back in, or run:)
newgrp docker

# Test
docker ps

Option B: Use sudo (Temporary)

sudo docker ps
sudo python3 launch_dotlake.py

Option C: Fix Docker socket permissions

sudo chmod 666 /var/run/docker.sock

2. Port Already in Use

Symptoms:

Error starting userland proxy: listen tcp 0.0.0.0:8080: bind: address already in use

Solutions:

Check what's using the ports:

# Check all ports used by DotLake
sudo lsof -i :3000  # Frontend
sudo lsof -i :8080  # API Sidecar
sudo lsof -i :8088  # Superset
sudo lsof -i :8501  # Ingest Service
sudo lsof -i :5432  # PostgreSQL

Kill processes using the ports:

# Kill process by PID
sudo kill -9 <PID>

# Or kill all processes on a port
sudo fuser -k 8080/tcp

Alternative: Use different ports

# In your config.yaml, you can modify ports in docker-compose files
# But this requires editing the docker-compose files directly

3. Configuration Validation Errors

Symptoms:

Configuration validation error:
  - chain.wss_endpoint: WSS endpoint must be a valid WebSocket URL

Common Solutions:

Invalid WSS Endpoint:

# ❌ Wrong
chain:
  wss_endpoint: https://polkadot-rpc.dwellir.com

# ✅ Correct
chain:
  wss_endpoint: wss://polkadot-rpc.dwellir.com

Invalid Block Range:

# ❌ Wrong (end < start)
ingest:
  mode: historical
  block_range:
    start: 1000
    end: 500

# ✅ Correct
ingest:
  mode: historical
  block_range:
    start: 500
    end: 1000

Multiple Database Configurations:

# ❌ Wrong (both enabled)
database:
  local:
    enabled: true
  external:
    enabled: true

# ✅ Correct
database:
  local:
    enabled: true
  external:
    enabled: false

4. Network Connectivity Issues

Symptoms:

Connection refused
WebSocket connection failed
Timeout waiting for response

Solutions:

Test WSS endpoint connectivity:

# Test WebSocket connection
wscat -c wss://polkadot-rpc.dwellir.com

# Test HTTP endpoint
curl -X POST -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"chain_getBlock","params":[],"id":1}' \
  https://polkadot-rpc.dwellir.com

Try alternative RPC endpoints:

# Alternative Polkadot endpoints
chain:
  wss_endpoint: wss://polkadot.api.onfinality.io/public-ws
  # or
  wss_endpoint: wss://polkadot-rpc-tn.dwellir.com
  # or
  wss_endpoint: wss://rpc.polkadot.io

Check firewall/proxy settings:

# Test if you can reach external services
curl https://google.com
curl https://polkadot-rpc.dwellir.com

5. Database Connection Issues

Symptoms:

psycopg2.OperationalError: could not connect to server
MySQL connection failed

Solutions:

Local PostgreSQL Issues:

# Check if PostgreSQL container is running
docker ps | grep postgres

# Check PostgreSQL logs
docker logs dotlake-postgres

# Restart PostgreSQL container
docker restart dotlake-postgres

# Check database connectivity
docker exec -it dotlake-postgres psql -U postgres -d dotlake -c "SELECT 1;"

External Database Issues:

# For local external database, use host.docker.internal
database:
  external:
    connection:
      host: host.docker.internal  # Not localhost
      port: 5432

Test external database connection:

# Test PostgreSQL
psql -h host.docker.internal -p 5432 -U username -d dotlake

# Test MySQL
mysql -h host.docker.internal -P 3306 -u username -p dotlake

6. No Data Being Ingested

Symptoms:

Frontend shows no blocks/events
Database tables are empty
No progress in logs

Diagnosis Steps:

Check ingest service logs:

docker logs -f dotlake-ingest

Look for these messages:

✅ Connected to database and created tables
✅ Processing block 12345678
✅ No new blocks to process. Retrying in 6 seconds.

Check API Sidecar connectivity:

# Test API Sidecar
curl http://localhost:8080/blocks/head

# Should return JSON like:
# {"number":"12345678","hash":"0x..."}

Check database tables:

# Connect to database
docker exec -it dotlake-postgres psql -U postgres -d dotlake

# Check if tables exist
\dt

# Check recent blocks
SELECT COUNT(*) FROM blocks;
SELECT * FROM blocks ORDER BY number DESC LIMIT 5;

Solutions:

If API Sidecar is not responding:

# Restart API Sidecar
docker restart dotlake-sidecar

# Check API Sidecar logs
docker logs dotlake-sidecar

If ingest service is not processing:

# Restart ingest service
docker restart dotlake-ingest

# Check for configuration errors
python3 test_config.py

7. Memory/Resource Issues

Symptoms:

Out of memory
Container killed
Service crashes repeatedly

Solutions:

Check system resources:

# Check memory usage
free -h

# Check disk space
df -h

# Check Docker resource usage
docker stats

Increase Docker resources:

Open Docker Desktop
Go to Settings → Resources
Increase Memory to 8GB+
Increase CPU to 4+

Optimize for low-resource systems:

# Use smaller block ranges for historical ingestion
ingest:
  mode: historical
  block_range:
    start: 1000000
    end: 1000100  # Small range for testing

8. Python Environment Issues

Symptoms:

ModuleNotFoundError: No module named 'pydantic'
ImportError: No module named 'yaml'

Solutions:

Recreate virtual environment:

# Remove existing environment
rm -rf venv

# Run simple launcher to recreate
python3 launch_dotlake.py

Manual virtual environment setup:

# Create virtual environment
python3 -m venv venv

# Activate
source venv/bin/activate  # Unix/Linux/macOS
# or
venv\Scripts\activate     # Windows

# Install dependencies
pip install -r requirements.txt

Check Python version:

python3 --version
# Should be 3.8 or higher

📋 Error Code Reference

Configuration Errors

Error	Cause	Solution
`WSS endpoint must be a valid WebSocket URL`	Invalid endpoint format	Use `wss://` not `https://`
`end_block must be greater than start_block`	Invalid block range	Ensure end > start
`Either local or external database must be enabled`	Database config conflict	Enable only one database type
`Database host must be a valid hostname`	Invalid host	Use `host.docker.internal` for local

Network Errors

Error	Cause	Solution
`Connection refused`	Service not running	Check if containers are up
`WebSocket connection failed`	Network/RPC issue	Try alternative RPC endpoint
`Timeout waiting for response`	Slow network/RPC	Use different RPC provider
`Address already in use`	Port conflict	Kill process using port

Database Errors

Error	Cause	Solution
`could not connect to server`	Database not accessible	Check host/port/credentials
`database does not exist`	Database not created	Check database name
`permission denied`	Wrong credentials	Verify username/password
`connection pool exhausted`	Too many connections	Restart database container

🔧 Advanced Troubleshooting

Debug Mode

Enable verbose logging:

# Set debug environment variable
export DOTLAKE_DEBUG=true

# Run with debug output
python3 launch_dotlake.py

Manual Service Testing

Test each service individually:

# Test API Sidecar
docker run --rm -p 8080:8080 parity/substrate-api-sidecar:latest

# Test PostgreSQL
docker run --rm -e POSTGRES_PASSWORD=test postgres:13

# Test ingest service
python3 ingest/main.py --help

Network Debugging

# Check Docker network
docker network ls
docker network inspect bridge

# Check container networking
docker exec dotlake-ingest ping google.com
docker exec dotlake-ingest curl http://dotlake-sidecar:8080/blocks/head

Database Debugging

# Check PostgreSQL configuration
docker exec dotlake-postgres cat /var/lib/postgresql/data/postgresql.conf

# Check MySQL configuration
docker exec dotlake-mysql cat /etc/mysql/my.cnf

# Test database performance
docker exec dotlake-postgres psql -U postgres -d dotlake -c "EXPLAIN ANALYZE SELECT * FROM blocks LIMIT 1000;"

📞 Getting Help

Before Asking for Help

Check this guide for your specific error
Search existing issues on GitHub
Collect diagnostic information:

# System information
uname -a
docker --version
python3 --version

# Configuration
cat config.yaml

# Service status
docker ps -a
docker logs dotlake-sidecar
docker logs dotlake-ingest
docker logs dotlake-postgres

# Network test
curl http://localhost:8080/blocks/head

Creating a Good Issue Report

When creating a GitHub issue, include:

Error message (exact text)
Configuration file (sanitized)
System information (OS, Docker version, etc.)
Steps to reproduce
Expected vs actual behavior
Logs (relevant portions)

Community Resources

GitHub Issues: Bug reports and feature requests
GitHub Discussions: General questions and community help
Documentation: Complete documentation index
Configuration Examples: Sample configurations

🎯 Prevention Tips

Best Practices

Always test configuration before running: python3 test_config.py
Use stable RPC endpoints for production
Monitor system resources during ingestion
Keep Docker updated to latest stable version
Use external databases for production deployments
Backup important data before major changes

Regular Maintenance

# Clean up unused Docker resources
docker system prune -f

# Update base images
docker pull parity/substrate-api-sidecar:latest
docker pull postgres:13
docker pull mysql:8

# Check for updates
git pull origin main

Still having issues? Create a detailed issue report with the diagnostic information above, and our community will help you resolve it!

Troubleshooting Guide - paritytech/dotlake-community GitHub Wiki

🔍 Quick Diagnosis

🚨 Common Issues

1. Docker Permission Errors

2. Port Already in Use

3. Configuration Validation Errors

4. Network Connectivity Issues

5. Database Connection Issues

6. No Data Being Ingested

7. Memory/Resource Issues

8. Python Environment Issues

📋 Error Code Reference

Configuration Errors

Network Errors

Database Errors

🔧 Advanced Troubleshooting

Debug Mode

Manual Service Testing

Network Debugging

Database Debugging

📞 Getting Help

Before Asking for Help

Creating a Good Issue Report

Community Resources

🎯 Prevention Tips

Best Practices

Regular Maintenance

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️