Monitoring & Analytics - georgi-dev215/openvpn-web-manager GitHub Wiki

Historical Data Storage

Database Tables

traffic_history - Individual session records 

CREATE TABLE traffic_history (
    id INTEGER PRIMARY KEY,
    client_name TEXT NOT NULL,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
    bytes_sent INTEGER DEFAULT 0,
    bytes_received INTEGER DEFAULT 0,
    duration_seconds INTEGER DEFAULT 0,
    session_start DATETIME,
    session_end DATETIME,
    real_address TEXT,
    virtual_address TEXT
);

client_stats - Aggregated client statistics 

CREATE TABLE client_stats (
    id INTEGER PRIMARY KEY,
    client_name TEXT UNIQUE NOT NULL,
    total_bytes_sent INTEGER DEFAULT 0,
    total_bytes_received INTEGER DEFAULT 0,
    total_duration_seconds INTEGER DEFAULT 0,
    session_count INTEGER DEFAULT 0,
    first_connection DATETIME,
    last_connection DATETIME,
    last_activity DATETIME,
    is_online INTEGER DEFAULT 0,
    current_session_start DATETIME
);

system_metrics - System performance history 

CREATE TABLE system_metrics (
    id INTEGER PRIMARY KEY,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
    cpu_percent REAL,
    memory_percent REAL,
    memory_available INTEGER,
    network_sent INTEGER,
    network_received INTEGER,
    active_connections INTEGER
);

System Metrics Collection

Automated Data Collection

Collection Schedule

  • Session Tracking: Every 30 seconds

  • System Metrics: Every 2 minutes

  • Client Statistics: Real-time on connection changes

Collected Metrics

System Performance 

system_metrics = {
    'cpu': {
        'percent': 45.2,        # CPU usage percentage
        'load_average': [1.2, 1.5, 1.8]  # 1, 5, 15 minute averages
    },
    'memory': {
        'total_gb': 8.0,        # Total RAM
        'used_gb': 5.2,         # Used RAM
        'usage_percent': 65.0,   # Memory usage percentage
        'available_gb': 2.8      # Available RAM
    },
    'network': {
        'upload_mbps': 12.5,     # Current upload speed
        'download_mbps': 25.8,   # Current download speed
        'bytes_sent': 1024000000,    # Total bytes sent
        'bytes_received': 2048000000  # Total bytes received
    }
}

Connection Statistics 

connection_stats = {
    'active_connections': 15,        # Currently connected clients
    'total_clients': 150,           # Total active clients  
    'total_bytes_sent': 50000000000,    # Server total sent
    'total_bytes_received': 75000000000  # Server total received
}

Log Monitoring

Available Log Types

OpenVPN Status Log

  • Purpose: Connection events and status changes

  • Content: Client connections, disconnections, errors

  • Location: /var/log/openvpn/status.log

OpenVPN Server Log

  • Purpose: Server operation and configuration events

  • Content: Server startup, configuration changes, errors

  • Location: /var/log/openvpn/openvpn.log

System Log

  • Purpose: System-level events and errors

  • Content: Service status, system errors, security events

  • Location: /var/log/syslog

Log Viewer Features

Real-Time Monitoring

  • Auto-refresh: Updates every 10 seconds

  • Line Limits: 25, 50, 100, 200, or 500 lines

  • Live Updates: New entries appear automatically

Log Filtering and Search

  • Text Filter: Search for specific terms

  • Highlight: Highlight matching text

  • Case Sensitivity: Optional case-sensitive search

Log Formatting

  • Syntax Highlighting: Different colors for log levels

  • ๐Ÿ”ด Error: Red background for error messages

  • ๐ŸŸก Warning: Yellow background for warnings

  • ๐ŸŸข Success: Green background for success messages

  • ๐Ÿ”ต Info: Blue background for info messages

Download and Export

  • Download Logs: Export current log view as text file

  • Line Numbers: Each log line numbered for reference

  • Timestamp Preservation: Original timestamps maintained

Log Analysis

Automatic Log Parsing

The system automatically identifies and highlights: - Connection Events: Client connect/disconnect - Authentication: Login success/failure - Errors: System and application errors - Performance: High resource usage warnings

Log Statistics

  • Error Count: Number of error entries

  • Warning Count: Number of warning entries

  • Success Rate: Connection success percentage

  • Peak Activity: Time periods with high activity

API Endpoints

Real-Time Data APIs

System Status

GET /api/status
Returns current server status and basic information.

Activity Data

GET /api/activity
Returns comprehensive activity data including: - Active connections list - Server statistics - System metrics - Network bandwidth

System Metrics

GET /api/system_metrics
Returns current system performance metrics.

Network Bandwidth

GET /api/network_bandwidth
Returns current network usage and speeds.

Historical Data APIs

Client Traffic History

GET /api/client_traffic/<client_name>?days=30
Returns historical traffic data for specific client.

Traffic Summary

GET /api/traffic_summary
Returns aggregated traffic data for all clients.

Log Access

GET /api/logs/<log_type>?lines=50
Returns log content with specified number of lines.

API Response Examples

Activity API Response

{
    "server_stats": {
        "active_connections": 15,
        "total_bytes_sent": 50000000000,
        "total_bytes_received": 75000000000
    },
    "active_connections": [
        {
            "name": "john_doe",
            "real_address": "203.0.113.45",
            "virtual_address": "10.8.0.6",
            "bytes_sent": "1310720",
            "bytes_received": "2621440",
            "connected_since": "2024-01-15 13:30:00",
            "duration_formatted": "2h 15m 30s"
        }
    ],
    "system_metrics": {
        "cpu": {"percent": 45.2},
        "memory": {"percent": 65.0}
    },
    "network_bandwidth": {
        "upload_mbps": 12.5,
        "download_mbps": 25.8
    }
}

Performance Optimization

Data Retention

Automatic Cleanup

  • Traffic History: Configurable retention period

  • System Metrics: Automatic archiving of old data

  • Log Rotation: System handles log file rotation

Database Optimization

-- Clean old session data (example for 30-day retention)
DELETE FROM traffic_history 
WHERE timestamp < datetime('now', '-30 days');

-- Rebuild database for optimal performance
VACUUM;

-- Check database size
SELECT page_count * page_size as size 
FROM pragma_page_count(), pragma_page_size();

Memory Management

Session Tracking

  • Active Sessions: Stored in memory for real-time access

  • Periodic Saves: Data written to database every 30 seconds

  • Memory Limits: Automatic cleanup of old session data

Chart Performance

  • Data Limits: Charts limited to 10-15 data points

  • Update Frequency: Balanced between accuracy and performance

  • Animation: Smooth transitions without performance impact

Troubleshooting

Common Issues

Dashboard Not Updating

# Check if monitoring is enabled
grep "track_client_sessions" /var/log/openvpn-manager/app.log

# Verify database connection
sqlite3 vpn_history.db "SELECT COUNT(*) FROM system_metrics;"

# Check API endpoints
curl http://localhost:8822/api/activity

Missing Traffic Data

# Check session tracking
ps aux | grep "python.*app.py"

# Verify database tables
sqlite3 vpn_history.db ".tables"

# Check for database errors
tail -f /var/log/openvpn-manager/app.log | grep -i database

High CPU Usage

# Monitor Chart.js performance
# Reduce chart update frequency in dashboard.html
# Limit number of data points displayed

# Check database queries
sqlite3 vpn_history.db "EXPLAIN QUERY PLAN SELECT * FROM traffic_history ORDER BY timestamp DESC LIMIT 100;"

Performance Monitoring

System Resource Usage

# Monitor application memory usage
ps -o pid,rss,vsz,comm -p $(pgrep -f "python.*app.py")

# Check database size
ls -lh vpn_history.db

# Monitor network usage
netstat -i

Database Performance

-- Check table sizes
SELECT name, COUNT(*) as rows FROM sqlite_master 
JOIN (SELECT name FROM sqlite_master WHERE type='table') 
ON 1=1 GROUP BY name;

-- Analyze query performance  
.timer on
SELECT client_name, SUM(bytes_sent), SUM(bytes_received) 
FROM traffic_history 
WHERE timestamp > datetime('now', '-7 days') 
GROUP BY client_name;

Best Practices

Data Management

  • Regular Backups: Backup vpn_history.db regularly

  • Data Retention: Set appropriate retention periods

  • Index Maintenance: Database indexes are automatically maintained

Performance Optimization

  • Monitor Resources: Keep an eye on CPU and memory usage

  • Chart Limits: Donโ€™t display too many data points simultaneously

  • Update Intervals: Balance between real-time updates and performance

Security Considerations

  • Log Access: Logs may contain sensitive information

  • API Security: All monitoring APIs require authentication

  • Data Privacy: Consider data retention policies for compliance

โš ๏ธ **GitHub.com Fallback** โš ๏ธ