Memory Technologies Production Limited BCC Memleak Sampled - antimetal/system-agent GitHub Wiki
BCC memleak with sampling provides practical memory leak detection for production environments through selective allocation tracing. This eBPF-based tool offers a balanced approach between diagnostic accuracy and system performance by sampling memory allocations at configurable rates.
Key Characteristics:
- eBPF-based malloc/free tracing with configurable sampling
- Part of BCC (BPF Compiler Collection) tools suite
- 10-30% overhead with appropriate sampling (vs 30-400% without)
- Tracks outstanding allocations with complete stack traces
- Production-suitable for periodic diagnostic runs
- Supports both user-space and kernel allocation tracking
- CPU Overhead: 10-30% with sampling rates of 100-1000 (1% to 0.1% of allocations)
- Memory Overhead: Moderate - proportional to sampling rate and allocation lifetime
- Latency Impact: 15-50% increase in allocation latency (sampled operations only)
-
Throughput Impact: Measured examples with sampling:
- MySQL: 8-15% throughput reduction (vs 33% with full tracing)
- Node.js APIs: 20-35% latency increase (vs 300% with full tracing)
- High-frequency apps: 40-70% latency increase (vs 200-400% with full tracing)
- Accuracy: High - statistical sampling provides representative leak detection
- False Positives: Low - genuine allocations without matching frees
- False Negatives: Moderate - may miss small, short-lived leaks
- Stack Trace Quality: Complete for sampled allocations
- Leak Size Detection: Excellent for significant leaks (>1KB), moderate for small leaks
- Linux Kernel: 4.6+ (eBPF uprobe support required)
- BCC Framework: Required for eBPF compilation and execution
- Debug Symbols: Recommended for meaningful stack traces
- Production Ready: Yes - with appropriate sampling configuration
BCC memleak sampling should be implemented as a Layer 3 diagnostic tool within the System-Agent architecture:
-
Trigger-Based Activation
memleak_sampling: enabled: true trigger_conditions: - memory_growth_rate: ">5MB/min" - rss_increase_threshold: ">100MB" - oom_risk_score: ">0.7" sampling_config: default_sample_rate: 1000 # 0.1% of allocations aggressive_sample_rate: 100 # 1% for urgent cases duration_limit: 300 # 5 minutes maximum
-
Adaptive Sampling Configuration
- Conservative (sample_rate=1000): For continuous light monitoring
- Balanced (sample_rate=500): For active investigation
- Aggressive (sample_rate=100): For urgent leak detection
- Size-filtered sampling: Focus on allocations >1KB for production
-
Integration Points
class MemleakSampledCollector: def __init__(self, config): self.sample_rate = config.get('sample_rate', 1000) self.min_size = config.get('min_size', 1024) self.duration_limit = config.get('duration_limit', 300) def should_trigger(self, metrics): return ( metrics.memory_growth_rate > self.growth_threshold or metrics.rss_increase > self.rss_threshold )
-
Duration-Limited Runs
- Maximum 5 minutes for production environments
- Maximum 15 minutes for staging environments
- Automatic termination with timeout protection
- Graceful shutdown with data preservation
# Basic sampling options
-s, --sample-rate RATE Record every Nth allocation (default: 1, no sampling)
-z, --min-size SIZE Capture allocations >= SIZE bytes
-Z, --max-size SIZE Capture allocations <= SIZE bytes
-o, --older MILLISEC Print allocations older than MILLISEC (default: 500ms)
# Process targeting
-p, --pid PID Trace specific process
-c, --command CMD Execute and trace command
# Output control
-a, --show-allocs Print every allocation/free
-t, --trace Print trace messages
--combined-only Use kernel-space statistics (lower overhead)
-T, --top COUNT Display top COUNT stacks (default: 10)
# Performance tuning
--stack-depth DEPTH Maximum stack trace depth (default: 16)
-O, --obj OBJECT Attach to allocations in specific object/library
# Conservative sampling for production monitoring
sudo memleak -p $PID -s 1000 -z 1024 -T 300
# Balanced sampling for active investigation
sudo memleak -p $PID -s 500 -o 1000 -T 600
# Aggressive sampling for urgent debugging
sudo memleak -p $PID -s 100 -a -T 180
# Size-focused sampling (large allocations only)
sudo memleak -p $PID -s 100 -z 4096 -T 300
# Combined statistics mode (lowest overhead)
sudo memleak -p $PID -s 1000 --combined-only -T 600
Standard Sampled Output:
Attaching to pid 12345, Ctrl+C to quit.
[...]
32768 bytes in 8 allocations from stack
0x7f8b4c000abc malloc+0x2c (/lib/x86_64-linux-gnu/libc.so.6)
0x55cb9c001234 allocate_buffer+0x45 (/usr/bin/myapp)
0x55cb9c002567 process_request+0x123 (/usr/bin/myapp)
0x55cb9c003890 main+0x234 (/usr/bin/myapp)
16384 bytes in 4 allocations from stack
0x7f8b4c000def calloc+0x1f (/lib/x86_64-linux-gnu/libc.so.6)
0x55cb9c004abc init_cache+0x67 (/usr/bin/myapp)
Interpretation Guide:
- Total bytes: Accumulated unfreed memory from this stack trace
- Allocation count: Number of unfreed allocations (before sampling adjustment)
- Stack trace: Call chain leading to allocation
- Sampling note: Actual leak size may be sample_rate × reported size
#!/usr/bin/env python3
# analyze_sampled_leaks.py
def analyze_sampled_output(output_file, sample_rate):
"""Analyze BCC memleak sampled output with rate adjustment."""
leaks = []
current_leak = None
with open(output_file, 'r') as f:
for line in f:
# Parse leak summary line
if re.match(r'^\d+ bytes in \d+ allocations', line):
if current_leak:
leaks.append(current_leak)
parts = line.split()
current_leak = {
'sampled_bytes': int(parts[0]),
'estimated_bytes': int(parts[0]) * sample_rate,
'sampled_count': int(parts[3]),
'estimated_count': int(parts[3]) * sample_rate,
'stack_trace': []
}
# Collect stack trace lines
elif current_leak and line.strip().startswith('0x'):
current_leak['stack_trace'].append(line.strip())
return leaks
def prioritize_leaks(leaks):
"""Prioritize leaks by estimated total size."""
return sorted(leaks, key=lambda x: x['estimated_bytes'], reverse=True)
-
Continuous Light Monitoring
# Run every 6 hours for 5 minutes */6 * * * * /usr/local/bin/memleak-monitor.sh
-
Triggered Investigations
# Triggered by memory growth alerts if [[ $RSS_GROWTH_MB -gt 100 ]]; then sudo memleak -p $PID -s 500 -z 1024 -T 300 > "/tmp/leak-$(date +%s).txt" fi
-
Scheduled Health Checks
# Weekly comprehensive sampling sudo memleak -p $PID -s 200 -o 2000 -T 900 > "weekly-leak-check.txt"
Environment: Node.js API server, 8GB RAM
Problem: 50MB/hour memory growth
Configuration: -s 1000 -z 1024 -T 300 (5-minute run)
Results:
- Overhead: 18% during sampling period
- Found: Large JSON parsing buffers not released
- Root cause: Missing explicit buffer cleanup in error paths
- Fix: Added proper cleanup in exception handlers
- Outcome: Memory growth reduced to <5MB/hour
Environment: PostgreSQL connection pool service
Problem: Gradual memory increase over days
Configuration: -s 500 -o 5000 -T 600 (10-minute run)
Results:
- Overhead: 25% during sampling period
- Found: Connection metadata structs accumulating
- Root cause: Connection timeout cleanup logic bug
- Fix: Corrected connection pool cleanup timing
- Outcome: Stable memory usage achieved
Environment: Python image processing microservice
Problem: Memory spikes during batch operations
Configuration: -s 100 -z 4096 -T 180 (3-minute run)
Results:
- Overhead: 35% during sampling period
- Found: Large image buffers retained between batches
- Root cause: Caching layer not respecting size limits
- Fix: Implemented LRU cache with memory limits
- Outcome: Predictable memory usage patterns
- Primary Repository: https://github.com/iovisor/bcc
-
Tool Source:
tools/memleak.py
(Python implementation) -
C Implementation:
libbpf-tools/memleak.c
(newer C version) -
Example Documentation:
tools/memleak_example.txt
-
Man Page:
man memleak-bpfcc
- eBPF Program: Uprobe attachment to libc allocation functions
- Sampling Logic: Configurable skip-counter in eBPF program
- Data Collection: BPF hash maps for allocation tracking
- Stack Walking: Kernel and user-space stack unwinding
- Symbol Resolution: Integration with system symbol tables
- memleak.c: Newer libbpf-based implementation with lower overhead
- profile: CPU profiling with memory allocation insights
- funclatency: Function call latency analysis for allocation functions
- trace: General-purpose tracing for custom allocation patterns
#!/bin/bash
# production-memleak-monitor.sh
set -euo pipefail
# Configuration
readonly PID="${1:-}"
readonly SAMPLE_RATE="${2:-1000}"
readonly DURATION="${3:-300}"
readonly MIN_SIZE="${4:-1024}"
readonly OUTPUT_DIR="/var/log/memory-monitoring"
# Validation
if [[ -z "$PID" ]]; then
echo "Usage: $0 <pid> [sample_rate] [duration] [min_size]"
exit 1
fi
if ! kill -0 "$PID" 2>/dev/null; then
echo "ERROR: Process $PID not found or not accessible"
exit 1
fi
# Ensure output directory exists
mkdir -p "$OUTPUT_DIR"
# Generate unique output filename
readonly TIMESTAMP=$(date +%Y%m%d_%H%M%S)
readonly OUTPUT_FILE="$OUTPUT_DIR/memleak_${PID}_${TIMESTAMP}.txt"
# Memory usage before tracing
echo "Pre-tracing memory usage:" > "$OUTPUT_FILE"
ps -p "$PID" -o pid,vsz,rss,comm >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"
# Run sampled tracing with timeout protection
echo "Starting sampled memory leak detection..."
echo "PID: $PID, Sample Rate: 1/$SAMPLE_RATE, Duration: ${DURATION}s"
timeout "${DURATION}s" sudo /usr/share/bcc/tools/memleak \
-p "$PID" \
-s "$SAMPLE_RATE" \
-z "$MIN_SIZE" \
-o 1000 \
--combined-only \
>> "$OUTPUT_FILE" 2>&1 || {
echo "Tracing completed (timeout or manual stop)"
}
# Memory usage after tracing
echo "" >> "$OUTPUT_FILE"
echo "Post-tracing memory usage:" >> "$OUTPUT_FILE"
ps -p "$PID" -o pid,vsz,rss,comm >> "$OUTPUT_FILE"
echo "Results saved to: $OUTPUT_FILE"
# Basic analysis
if grep -q "bytes in.*allocations" "$OUTPUT_FILE"; then
echo "ALERT: Potential memory leaks detected"
grep "bytes in.*allocations" "$OUTPUT_FILE" | head -5
else
echo "No significant leaks detected in sampling period"
fi
#!/usr/bin/env python3
# memleak_sampled_analyzer.py
import subprocess
import time
import json
import re
from typing import Dict, List, Optional
from dataclasses import dataclass
@dataclass
class LeakInfo:
bytes_leaked: int
allocation_count: int
estimated_total_bytes: int
estimated_total_count: int
stack_trace: List[str]
avg_allocation_size: int
class MemleakSampledAnalyzer:
def __init__(self, sample_rate: int = 1000, min_size: int = 1024):
self.sample_rate = sample_rate
self.min_size = min_size
self.duration_limit = 300 # 5 minutes maximum
def run_analysis(self, pid: int, duration: int = 300) -> List[LeakInfo]:
"""Run sampled memory leak analysis on target process."""
if duration > self.duration_limit:
duration = self.duration_limit
cmd = [
'sudo', '/usr/share/bcc/tools/memleak',
'-p', str(pid),
'-s', str(self.sample_rate),
'-z', str(self.min_size),
'-o', '1000', # 1 second minimum age
'--combined-only' # Use kernel statistics for performance
]
try:
# Run with timeout protection
result = subprocess.run(
cmd,
timeout=duration,
capture_output=True,
text=True,
check=False
)
return self._parse_output(result.stdout)
except subprocess.TimeoutExpired:
print(f"Sampling completed after {duration} seconds")
return []
except Exception as e:
print(f"Error running memleak: {e}")
return []
def _parse_output(self, output: str) -> List[LeakInfo]:
"""Parse memleak output and extract leak information."""
leaks = []
lines = output.split('\n')
i = 0
while i < len(lines):
line = lines[i]
# Look for leak summary lines
match = re.match(r'^(\d+) bytes in (\d+) allocations from stack', line)
if match:
sampled_bytes = int(match.group(1))
sampled_count = int(match.group(2))
# Estimate actual totals based on sampling rate
estimated_bytes = sampled_bytes * self.sample_rate
estimated_count = sampled_count * self.sample_rate
# Collect stack trace
stack_trace = []
i += 1
while i < len(lines) and lines[i].strip().startswith('0x'):
stack_trace.append(lines[i].strip())
i += 1
leaks.append(LeakInfo(
bytes_leaked=sampled_bytes,
allocation_count=sampled_count,
estimated_total_bytes=estimated_bytes,
estimated_total_count=estimated_count,
stack_trace=stack_trace,
avg_allocation_size=estimated_bytes // max(estimated_count, 1)
))
continue
i += 1
return sorted(leaks, key=lambda x: x.estimated_total_bytes, reverse=True)
def generate_alert(self, leaks: List[LeakInfo], threshold_mb: int = 10) -> Optional[Dict]:
"""Generate alert if significant leaks detected."""
if not leaks:
return None
total_estimated_leaked = sum(leak.estimated_total_bytes for leak in leaks)
if total_estimated_leaked < threshold_mb * 1024 * 1024:
return None
return {
'severity': 'high' if total_estimated_leaked > 100*1024*1024 else 'medium',
'total_leaked_mb': total_estimated_leaked / (1024 * 1024),
'leak_sites': len(leaks),
'top_leak': {
'estimated_bytes': leaks[0].estimated_total_bytes,
'stack_preview': leaks[0].stack_trace[:3]
},
'sampling_info': {
'sample_rate': self.sample_rate,
'accuracy_note': f'Estimates based on 1/{self.sample_rate} sampling'
}
}
# Usage example
if __name__ == "__main__":
analyzer = MemleakSampledAnalyzer(sample_rate=500, min_size=1024)
pid = 12345 # Target process
print(f"Starting sampled leak analysis for PID {pid}")
leaks = analyzer.run_analysis(pid, duration=300)
if leaks:
print(f"Found {len(leaks)} potential leak sites")
for i, leak in enumerate(leaks[:5], 1):
print(f"{i}. Estimated {leak.estimated_total_bytes:,} bytes "
f"in {leak.estimated_total_count:,} allocations")
alert = analyzer.generate_alert(leaks)
if alert:
print("ALERT:", json.dumps(alert, indent=2))
# systemagent_memleak_integration.py
class SystemAgentMemleakIntegration:
def __init__(self, config):
self.config = config
self.analyzer = MemleakSampledAnalyzer(
sample_rate=config.get('sample_rate', 1000),
min_size=config.get('min_size', 1024)
)
def should_trigger_sampling(self, process_metrics: Dict) -> bool:
"""Determine if memory leak sampling should be triggered."""
# Memory growth rate check
growth_rate = process_metrics.get('memory_growth_mb_per_hour', 0)
if growth_rate > 50: # >50MB/hour growth
return True
# RSS increase threshold
rss_increase = process_metrics.get('rss_increase_mb', 0)
if rss_increase > 100: # >100MB increase
return True
# OOM risk assessment
oom_risk = process_metrics.get('oom_risk_score', 0)
if oom_risk > 0.7: # High OOM risk
return True
return False
def execute_sampling(self, pid: int) -> Dict:
"""Execute memory leak sampling and return results."""
start_time = time.time()
# Adjust sampling rate based on allocation frequency
alloc_freq = self._estimate_allocation_frequency(pid)
sample_rate = self._calculate_optimal_sampling_rate(alloc_freq)
self.analyzer.sample_rate = sample_rate
leaks = self.analyzer.run_analysis(pid, duration=300)
alert = self.analyzer.generate_alert(leaks)
return {
'timestamp': start_time,
'pid': pid,
'sample_rate': sample_rate,
'duration': time.time() - start_time,
'leaks_detected': len(leaks),
'alert': alert,
'raw_leaks': [leak.__dict__ for leak in leaks[:10]] # Top 10
}
def calculate_optimal_sampling_rate(allocation_frequency: int) -> int:
"""Calculate optimal sampling rate based on allocation patterns."""
# Base sampling rates by allocation frequency
if allocation_frequency < 1000: # <1K allocs/sec
return 100 # 1% sampling
elif allocation_frequency < 10000: # <10K allocs/sec
return 500 # 0.2% sampling
elif allocation_frequency < 50000: # <50K allocs/sec
return 1000 # 0.1% sampling
else: # >50K allocs/sec
return 2000 # 0.05% sampling
# Filter configurations for different scenarios
# Large object leak detection (production safe)
memleak -p $PID -s 1000 -z 4096 -T 300
# Medium allocation monitoring (staging)
memleak -p $PID -s 500 -z 1024 -Z 1048576 -T 600
# Small allocation debugging (development only)
memleak -p $PID -s 100 -z 64 -T 180
#!/bin/bash
# protected-memleak-runner.sh
readonly MAX_PRODUCTION_DURATION=300 # 5 minutes
readonly MAX_STAGING_DURATION=900 # 15 minutes
readonly ENVIRONMENT="${ENVIRONMENT:-production}"
# Set duration limits based on environment
if [[ "$ENVIRONMENT" == "production" ]]; then
MAX_DURATION=$MAX_PRODUCTION_DURATION
else
MAX_DURATION=$MAX_STAGING_DURATION
fi
# Run with automatic termination
timeout "$MAX_DURATION" sudo memleak \
-p "$PID" \
-s "$SAMPLE_RATE" \
-z "$MIN_SIZE" \
--combined-only \
> "$OUTPUT_FILE" 2>&1 || {
echo "Sampling terminated after ${MAX_DURATION}s limit"
}
# monitoring-triggers.yaml
memory_leak_triggers:
# Threshold-based triggers
rss_growth_threshold: 100MB # RSS growth in 1 hour
memory_growth_rate: 50MB/hour # Sustained growth rate
oom_risk_score: 0.7 # Predictive OOM risk
# Pattern-based triggers
allocation_rate_spike: 5x # 5x increase in allocation rate
fragmentation_increase: 30% # Memory fragmentation increase
# Time-based triggers
scheduled_health_check: "0 2 * * 0" # Weekly at 2 AM Sunday
# Manual triggers
incident_response: true # On-demand for incidents
pre_deployment_check: true # Before major deployments
Environment | Max Duration | Sample Rate | Trigger Frequency |
---|---|---|---|
Production | 5 minutes | 1000-2000 | Incident-only |
Staging | 15 minutes | 500-1000 | Daily/Weekly |
Development | 30 minutes | 100-500 | On-demand |
CI/CD | 10 minutes | 200-500 | Per build |
# overhead_monitor.py
class OverheadMonitor:
def __init__(self):
self.baseline_metrics = {}
self.tracing_metrics = {}
def capture_baseline(self, pid: int):
"""Capture performance metrics before tracing."""
self.baseline_metrics = {
'cpu_usage': self._get_cpu_usage(pid),
'memory_usage': self._get_memory_usage(pid),
'response_time': self._get_response_time(pid)
}
def monitor_overhead(self, pid: int) -> Dict:
"""Monitor overhead during tracing."""
current_metrics = {
'cpu_usage': self._get_cpu_usage(pid),
'memory_usage': self._get_memory_usage(pid),
'response_time': self._get_response_time(pid)
}
overhead = {}
for metric, current_value in current_metrics.items():
baseline_value = self.baseline_metrics.get(metric, 0)
if baseline_value > 0:
overhead[f'{metric}_overhead_percent'] = \
((current_value - baseline_value) / baseline_value) * 100
return overhead
def should_terminate_early(self, overhead: Dict) -> bool:
"""Determine if tracing should be terminated due to high overhead."""
# Terminate if CPU overhead exceeds 50%
if overhead.get('cpu_usage_overhead_percent', 0) > 50:
return True
# Terminate if response time degrades >100%
if overhead.get('response_time_overhead_percent', 0) > 100:
return True
return False
Metric | Full Tracing | Sampled (1000:1) | Sampled (500:1) | Sampled (100:1) |
---|---|---|---|---|
CPU Overhead | 30-400% | 10-15% | 15-25% | 25-35% |
Memory Overhead | Very High | Low | Moderate | Moderate-High |
Allocation Latency | +100-400% | +5-15% | +10-25% | +20-40% |
Detection Accuracy | 100% | 85-95% | 90-98% | 95-99% |
Production Suitable | No | Yes | Limited | Development Only |
Full Tracing Advantages:
- Captures every allocation and deallocation
- Perfect accuracy for leak detection
- Complete allocation timeline
- No statistical estimation required
Sampled Tracing Trade-offs:
- May miss very small, short-lived leaks
- Requires statistical interpretation of results
- Less precise leak size estimates
- Potential for sampling bias in certain allocation patterns
Choose Full Tracing When:
✓ Development environment debugging
✓ Reproducing known leaks with minimal performance requirements
✓ Complete accuracy is essential
✓ Short-duration debugging sessions (<5 minutes)
Choose Sampled Tracing When:
✓ Production environment monitoring needed
✓ Long-duration analysis required (>5 minutes)
✓ System performance must be preserved
✓ Statistical leak detection is sufficient
✓ Continuous or periodic monitoring
# Start with conservative sampling
sudo memleak -p $PID -s 2000 -z 2048 -T 300
# If no leaks found, increase sensitivity
sudo memleak -p $PID -s 1000 -z 1024 -T 300
# If leaks detected, increase sampling for better accuracy
sudo memleak -p $PID -s 500 -z 1024 -T 180
# For final confirmation, brief full tracing (development only)
sudo memleak -p $PID -z 1024 -T 60 # No sampling, 1 minute max
# Target specific memory allocators
sudo memleak -p $PID -s 1000 -O libtcmalloc.so.4 # TCMalloc
sudo memleak -p $PID -s 1000 -O libjemalloc.so.2 # jemalloc
sudo memleak -p $PID -s 1000 -O libmimalloc.so # mimalloc
# User-space only (default)
sudo memleak -p $PID -s 1000
# Kernel allocations only
sudo memleak -K -s 500
# Combined user and kernel (higher overhead)
sudo memleak -p $PID -K -s 2000 --combined-only
# Different sampling rates for different allocation sizes
# Large allocations (>64KB) - sample more aggressively
sudo memleak -p $PID -s 100 -z 65536 -T 300 > large_allocs.txt &
# Medium allocations (1KB-64KB) - moderate sampling
sudo memleak -p $PID -s 1000 -z 1024 -Z 65535 -T 300 > medium_allocs.txt &
# Small allocations (<1KB) - light sampling only in development
sudo memleak -p $PID -s 5000 -Z 1023 -T 180 > small_allocs.txt &
- Use sampling rates ≥500 (≤0.2% of allocations)
- Set minimum size filters ≥1KB for production
- Limit duration to ≤5 minutes in production
- Use
--combined-only
flag for better performance - Monitor system overhead during sampling
- Have automatic termination safeguards
- Schedule during low-traffic periods when possible
- Size filtering to focus on significant allocations
- Appropriate sampling rate for allocation frequency
- Duration limits with timeout protection
- Combined statistics mode for reduced overhead
- Stack depth limits for faster collection
- Automated triggering based on memory growth patterns
- Statistical adjustment for sampling rate in alerts
- Integration with existing monitoring infrastructure
- Escalation procedures for confirmed leaks
- Documentation of sampling methodology for incident response
- BCC memleak (Full Tracing) - Complete tracing approach
- Hardware PMC - Hardware performance counter monitoring
- jemalloc Profiling - Alternative allocator profiling
- Memory Leak Detection Comparison Matrix - Tool comparison
- Continuous Memory Monitoring eBPF Design - System architecture