Memory Technologies Production Limited BCC Memleak Sampled - antimetal/system-agent GitHub Wiki

BCC memleak (Sampled)

Overview

BCC memleak with sampling provides practical memory leak detection for production environments through selective allocation tracing. This eBPF-based tool offers a balanced approach between diagnostic accuracy and system performance by sampling memory allocations at configurable rates.

Key Characteristics:

  • eBPF-based malloc/free tracing with configurable sampling
  • Part of BCC (BPF Compiler Collection) tools suite
  • 10-30% overhead with appropriate sampling (vs 30-400% without)
  • Tracks outstanding allocations with complete stack traces
  • Production-suitable for periodic diagnostic runs
  • Supports both user-space and kernel allocation tracking

Performance Characteristics

Overhead Analysis

  • CPU Overhead: 10-30% with sampling rates of 100-1000 (1% to 0.1% of allocations)
  • Memory Overhead: Moderate - proportional to sampling rate and allocation lifetime
  • Latency Impact: 15-50% increase in allocation latency (sampled operations only)
  • Throughput Impact: Measured examples with sampling:
    • MySQL: 8-15% throughput reduction (vs 33% with full tracing)
    • Node.js APIs: 20-35% latency increase (vs 300% with full tracing)
    • High-frequency apps: 40-70% latency increase (vs 200-400% with full tracing)

Accuracy Metrics

  • Accuracy: High - statistical sampling provides representative leak detection
  • False Positives: Low - genuine allocations without matching frees
  • False Negatives: Moderate - may miss small, short-lived leaks
  • Stack Trace Quality: Complete for sampled allocations
  • Leak Size Detection: Excellent for significant leaks (>1KB), moderate for small leaks

Platform Requirements

  • Linux Kernel: 4.6+ (eBPF uprobe support required)
  • BCC Framework: Required for eBPF compilation and execution
  • Debug Symbols: Recommended for meaningful stack traces
  • Production Ready: Yes - with appropriate sampling configuration

System-Agent Implementation Plan

Layer 3 Integration Strategy

BCC memleak sampling should be implemented as a Layer 3 diagnostic tool within the System-Agent architecture:

  1. Trigger-Based Activation

    memleak_sampling:
      enabled: true
      trigger_conditions:
        - memory_growth_rate: ">5MB/min"
        - rss_increase_threshold: ">100MB"
        - oom_risk_score: ">0.7"
      sampling_config:
        default_sample_rate: 1000  # 0.1% of allocations
        aggressive_sample_rate: 100  # 1% for urgent cases
        duration_limit: 300  # 5 minutes maximum
  2. Adaptive Sampling Configuration

    • Conservative (sample_rate=1000): For continuous light monitoring
    • Balanced (sample_rate=500): For active investigation
    • Aggressive (sample_rate=100): For urgent leak detection
    • Size-filtered sampling: Focus on allocations >1KB for production
  3. Integration Points

    class MemleakSampledCollector:
        def __init__(self, config):
            self.sample_rate = config.get('sample_rate', 1000)
            self.min_size = config.get('min_size', 1024)
            self.duration_limit = config.get('duration_limit', 300)
            
        def should_trigger(self, metrics):
            return (
                metrics.memory_growth_rate > self.growth_threshold or
                metrics.rss_increase > self.rss_threshold
            )
  4. Duration-Limited Runs

    • Maximum 5 minutes for production environments
    • Maximum 15 minutes for staging environments
    • Automatic termination with timeout protection
    • Graceful shutdown with data preservation

Tool Usage

Command Line Options

# Basic sampling options
-s, --sample-rate RATE    Record every Nth allocation (default: 1, no sampling)
-z, --min-size SIZE       Capture allocations >= SIZE bytes
-Z, --max-size SIZE       Capture allocations <= SIZE bytes
-o, --older MILLISEC      Print allocations older than MILLISEC (default: 500ms)

# Process targeting
-p, --pid PID            Trace specific process
-c, --command CMD        Execute and trace command

# Output control
-a, --show-allocs        Print every allocation/free
-t, --trace              Print trace messages
--combined-only          Use kernel-space statistics (lower overhead)
-T, --top COUNT          Display top COUNT stacks (default: 10)

# Performance tuning
--stack-depth DEPTH      Maximum stack trace depth (default: 16)
-O, --obj OBJECT         Attach to allocations in specific object/library

Practical Sampling Examples

# Conservative sampling for production monitoring
sudo memleak -p $PID -s 1000 -z 1024 -T 300

# Balanced sampling for active investigation
sudo memleak -p $PID -s 500 -o 1000 -T 600

# Aggressive sampling for urgent debugging
sudo memleak -p $PID -s 100 -a -T 180

# Size-focused sampling (large allocations only)
sudo memleak -p $PID -s 100 -z 4096 -T 300

# Combined statistics mode (lowest overhead)
sudo memleak -p $PID -s 1000 --combined-only -T 600

Output Interpretation

Standard Sampled Output:

Attaching to pid 12345, Ctrl+C to quit.
[...]
32768 bytes in 8 allocations from stack
    0x7f8b4c000abc    malloc+0x2c (/lib/x86_64-linux-gnu/libc.so.6)
    0x55cb9c001234    allocate_buffer+0x45 (/usr/bin/myapp)
    0x55cb9c002567    process_request+0x123 (/usr/bin/myapp)
    0x55cb9c003890    main+0x234 (/usr/bin/myapp)

16384 bytes in 4 allocations from stack
    0x7f8b4c000def    calloc+0x1f (/lib/x86_64-linux-gnu/libc.so.6)
    0x55cb9c004abc    init_cache+0x67 (/usr/bin/myapp)

Interpretation Guide:

  • Total bytes: Accumulated unfreed memory from this stack trace
  • Allocation count: Number of unfreed allocations (before sampling adjustment)
  • Stack trace: Call chain leading to allocation
  • Sampling note: Actual leak size may be sample_rate × reported size

Stack Trace Analysis

#!/usr/bin/env python3
# analyze_sampled_leaks.py

def analyze_sampled_output(output_file, sample_rate):
    """Analyze BCC memleak sampled output with rate adjustment."""
    
    leaks = []
    current_leak = None
    
    with open(output_file, 'r') as f:
        for line in f:
            # Parse leak summary line
            if re.match(r'^\d+ bytes in \d+ allocations', line):
                if current_leak:
                    leaks.append(current_leak)
                
                parts = line.split()
                current_leak = {
                    'sampled_bytes': int(parts[0]),
                    'estimated_bytes': int(parts[0]) * sample_rate,
                    'sampled_count': int(parts[3]),
                    'estimated_count': int(parts[3]) * sample_rate,
                    'stack_trace': []
                }
            
            # Collect stack trace lines
            elif current_leak and line.strip().startswith('0x'):
                current_leak['stack_trace'].append(line.strip())
    
    return leaks

def prioritize_leaks(leaks):
    """Prioritize leaks by estimated total size."""
    return sorted(leaks, key=lambda x: x['estimated_bytes'], reverse=True)

Production Deployments

Deployment Scenarios

  1. Continuous Light Monitoring

    # Run every 6 hours for 5 minutes
    */6 * * * * /usr/local/bin/memleak-monitor.sh
  2. Triggered Investigations

    # Triggered by memory growth alerts
    if [[ $RSS_GROWTH_MB -gt 100 ]]; then
        sudo memleak -p $PID -s 500 -z 1024 -T 300 > "/tmp/leak-$(date +%s).txt"
    fi
  3. Scheduled Health Checks

    # Weekly comprehensive sampling
    sudo memleak -p $PID -s 200 -o 2000 -T 900 > "weekly-leak-check.txt"

Case Studies of Successful Leak Detection

Case Study 1: Web Server Memory Growth

Environment: Node.js API server, 8GB RAM
Problem: 50MB/hour memory growth
Configuration: -s 1000 -z 1024 -T 300 (5-minute run)

Results:
- Overhead: 18% during sampling period
- Found: Large JSON parsing buffers not released
- Root cause: Missing explicit buffer cleanup in error paths
- Fix: Added proper cleanup in exception handlers
- Outcome: Memory growth reduced to <5MB/hour

Case Study 2: Database Connection Pool Leak

Environment: PostgreSQL connection pool service
Problem: Gradual memory increase over days
Configuration: -s 500 -o 5000 -T 600 (10-minute run)

Results:
- Overhead: 25% during sampling period
- Found: Connection metadata structs accumulating
- Root cause: Connection timeout cleanup logic bug
- Fix: Corrected connection pool cleanup timing
- Outcome: Stable memory usage achieved

Case Study 3: Image Processing Service

Environment: Python image processing microservice
Problem: Memory spikes during batch operations
Configuration: -s 100 -z 4096 -T 180 (3-minute run)

Results:
- Overhead: 35% during sampling period
- Found: Large image buffers retained between batches
- Root cause: Caching layer not respecting size limits
- Fix: Implemented LRU cache with memory limits
- Outcome: Predictable memory usage patterns

Repository & Documentation

Official Sources

  • Primary Repository: https://github.com/iovisor/bcc
  • Tool Source: tools/memleak.py (Python implementation)
  • C Implementation: libbpf-tools/memleak.c (newer C version)
  • Example Documentation: tools/memleak_example.txt
  • Man Page: man memleak-bpfcc

Implementation Architecture

  • eBPF Program: Uprobe attachment to libc allocation functions
  • Sampling Logic: Configurable skip-counter in eBPF program
  • Data Collection: BPF hash maps for allocation tracking
  • Stack Walking: Kernel and user-space stack unwinding
  • Symbol Resolution: Integration with system symbol tables

Related Tools

  • memleak.c: Newer libbpf-based implementation with lower overhead
  • profile: CPU profiling with memory allocation insights
  • funclatency: Function call latency analysis for allocation functions
  • trace: General-purpose tracing for custom allocation patterns

Code Examples

Production Monitoring Script

#!/bin/bash
# production-memleak-monitor.sh

set -euo pipefail

# Configuration
readonly PID="${1:-}"
readonly SAMPLE_RATE="${2:-1000}"
readonly DURATION="${3:-300}"
readonly MIN_SIZE="${4:-1024}"
readonly OUTPUT_DIR="/var/log/memory-monitoring"

# Validation
if [[ -z "$PID" ]]; then
    echo "Usage: $0 <pid> [sample_rate] [duration] [min_size]"
    exit 1
fi

if ! kill -0 "$PID" 2>/dev/null; then
    echo "ERROR: Process $PID not found or not accessible"
    exit 1
fi

# Ensure output directory exists
mkdir -p "$OUTPUT_DIR"

# Generate unique output filename
readonly TIMESTAMP=$(date +%Y%m%d_%H%M%S)
readonly OUTPUT_FILE="$OUTPUT_DIR/memleak_${PID}_${TIMESTAMP}.txt"

# Memory usage before tracing
echo "Pre-tracing memory usage:" > "$OUTPUT_FILE"
ps -p "$PID" -o pid,vsz,rss,comm >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"

# Run sampled tracing with timeout protection
echo "Starting sampled memory leak detection..."
echo "PID: $PID, Sample Rate: 1/$SAMPLE_RATE, Duration: ${DURATION}s"

timeout "${DURATION}s" sudo /usr/share/bcc/tools/memleak \
    -p "$PID" \
    -s "$SAMPLE_RATE" \
    -z "$MIN_SIZE" \
    -o 1000 \
    --combined-only \
    >> "$OUTPUT_FILE" 2>&1 || {
    echo "Tracing completed (timeout or manual stop)"
}

# Memory usage after tracing
echo "" >> "$OUTPUT_FILE"
echo "Post-tracing memory usage:" >> "$OUTPUT_FILE"
ps -p "$PID" -o pid,vsz,rss,comm >> "$OUTPUT_FILE"

echo "Results saved to: $OUTPUT_FILE"

# Basic analysis
if grep -q "bytes in.*allocations" "$OUTPUT_FILE"; then
    echo "ALERT: Potential memory leaks detected"
    grep "bytes in.*allocations" "$OUTPUT_FILE" | head -5
else
    echo "No significant leaks detected in sampling period"
fi

Python Integration Code

#!/usr/bin/env python3
# memleak_sampled_analyzer.py

import subprocess
import time
import json
import re
from typing import Dict, List, Optional
from dataclasses import dataclass

@dataclass
class LeakInfo:
    bytes_leaked: int
    allocation_count: int
    estimated_total_bytes: int
    estimated_total_count: int
    stack_trace: List[str]
    avg_allocation_size: int

class MemleakSampledAnalyzer:
    def __init__(self, sample_rate: int = 1000, min_size: int = 1024):
        self.sample_rate = sample_rate
        self.min_size = min_size
        self.duration_limit = 300  # 5 minutes maximum
        
    def run_analysis(self, pid: int, duration: int = 300) -> List[LeakInfo]:
        """Run sampled memory leak analysis on target process."""
        
        if duration > self.duration_limit:
            duration = self.duration_limit
            
        cmd = [
            'sudo', '/usr/share/bcc/tools/memleak',
            '-p', str(pid),
            '-s', str(self.sample_rate),
            '-z', str(self.min_size),
            '-o', '1000',  # 1 second minimum age
            '--combined-only'  # Use kernel statistics for performance
        ]
        
        try:
            # Run with timeout protection
            result = subprocess.run(
                cmd,
                timeout=duration,
                capture_output=True,
                text=True,
                check=False
            )
            
            return self._parse_output(result.stdout)
            
        except subprocess.TimeoutExpired:
            print(f"Sampling completed after {duration} seconds")
            return []
        except Exception as e:
            print(f"Error running memleak: {e}")
            return []
    
    def _parse_output(self, output: str) -> List[LeakInfo]:
        """Parse memleak output and extract leak information."""
        leaks = []
        lines = output.split('\n')
        
        i = 0
        while i < len(lines):
            line = lines[i]
            
            # Look for leak summary lines
            match = re.match(r'^(\d+) bytes in (\d+) allocations from stack', line)
            if match:
                sampled_bytes = int(match.group(1))
                sampled_count = int(match.group(2))
                
                # Estimate actual totals based on sampling rate
                estimated_bytes = sampled_bytes * self.sample_rate
                estimated_count = sampled_count * self.sample_rate
                
                # Collect stack trace
                stack_trace = []
                i += 1
                while i < len(lines) and lines[i].strip().startswith('0x'):
                    stack_trace.append(lines[i].strip())
                    i += 1
                
                leaks.append(LeakInfo(
                    bytes_leaked=sampled_bytes,
                    allocation_count=sampled_count,
                    estimated_total_bytes=estimated_bytes,
                    estimated_total_count=estimated_count,
                    stack_trace=stack_trace,
                    avg_allocation_size=estimated_bytes // max(estimated_count, 1)
                ))
                continue
            
            i += 1
        
        return sorted(leaks, key=lambda x: x.estimated_total_bytes, reverse=True)
    
    def generate_alert(self, leaks: List[LeakInfo], threshold_mb: int = 10) -> Optional[Dict]:
        """Generate alert if significant leaks detected."""
        
        if not leaks:
            return None
            
        total_estimated_leaked = sum(leak.estimated_total_bytes for leak in leaks)
        
        if total_estimated_leaked < threshold_mb * 1024 * 1024:
            return None
            
        return {
            'severity': 'high' if total_estimated_leaked > 100*1024*1024 else 'medium',
            'total_leaked_mb': total_estimated_leaked / (1024 * 1024),
            'leak_sites': len(leaks),
            'top_leak': {
                'estimated_bytes': leaks[0].estimated_total_bytes,
                'stack_preview': leaks[0].stack_trace[:3]
            },
            'sampling_info': {
                'sample_rate': self.sample_rate,
                'accuracy_note': f'Estimates based on 1/{self.sample_rate} sampling'
            }
        }

# Usage example
if __name__ == "__main__":
    analyzer = MemleakSampledAnalyzer(sample_rate=500, min_size=1024)
    pid = 12345  # Target process
    
    print(f"Starting sampled leak analysis for PID {pid}")
    leaks = analyzer.run_analysis(pid, duration=300)
    
    if leaks:
        print(f"Found {len(leaks)} potential leak sites")
        for i, leak in enumerate(leaks[:5], 1):
            print(f"{i}. Estimated {leak.estimated_total_bytes:,} bytes "
                  f"in {leak.estimated_total_count:,} allocations")
    
    alert = analyzer.generate_alert(leaks)
    if alert:
        print("ALERT:", json.dumps(alert, indent=2))

Automated Analysis Integration

# systemagent_memleak_integration.py

class SystemAgentMemleakIntegration:
    def __init__(self, config):
        self.config = config
        self.analyzer = MemleakSampledAnalyzer(
            sample_rate=config.get('sample_rate', 1000),
            min_size=config.get('min_size', 1024)
        )
    
    def should_trigger_sampling(self, process_metrics: Dict) -> bool:
        """Determine if memory leak sampling should be triggered."""
        
        # Memory growth rate check
        growth_rate = process_metrics.get('memory_growth_mb_per_hour', 0)
        if growth_rate > 50:  # >50MB/hour growth
            return True
            
        # RSS increase threshold
        rss_increase = process_metrics.get('rss_increase_mb', 0)
        if rss_increase > 100:  # >100MB increase
            return True
            
        # OOM risk assessment
        oom_risk = process_metrics.get('oom_risk_score', 0)
        if oom_risk > 0.7:  # High OOM risk
            return True
            
        return False
    
    def execute_sampling(self, pid: int) -> Dict:
        """Execute memory leak sampling and return results."""
        
        start_time = time.time()
        
        # Adjust sampling rate based on allocation frequency
        alloc_freq = self._estimate_allocation_frequency(pid)
        sample_rate = self._calculate_optimal_sampling_rate(alloc_freq)
        
        self.analyzer.sample_rate = sample_rate
        
        leaks = self.analyzer.run_analysis(pid, duration=300)
        alert = self.analyzer.generate_alert(leaks)
        
        return {
            'timestamp': start_time,
            'pid': pid,
            'sample_rate': sample_rate,
            'duration': time.time() - start_time,
            'leaks_detected': len(leaks),
            'alert': alert,
            'raw_leaks': [leak.__dict__ for leak in leaks[:10]]  # Top 10
        }

Optimization Strategies

Sampling Rate Selection

def calculate_optimal_sampling_rate(allocation_frequency: int) -> int:
    """Calculate optimal sampling rate based on allocation patterns."""
    
    # Base sampling rates by allocation frequency
    if allocation_frequency < 1000:      # <1K allocs/sec
        return 100    # 1% sampling
    elif allocation_frequency < 10000:   # <10K allocs/sec
        return 500    # 0.2% sampling
    elif allocation_frequency < 50000:   # <50K allocs/sec
        return 1000   # 0.1% sampling
    else:                               # >50K allocs/sec
        return 2000   # 0.05% sampling

Size Filtering Strategy

# Filter configurations for different scenarios

# Large object leak detection (production safe)
memleak -p $PID -s 1000 -z 4096 -T 300

# Medium allocation monitoring (staging)
memleak -p $PID -s 500 -z 1024 -Z 1048576 -T 600

# Small allocation debugging (development only)
memleak -p $PID -s 100 -z 64 -T 180

Duration Limits and Protection

#!/bin/bash
# protected-memleak-runner.sh

readonly MAX_PRODUCTION_DURATION=300    # 5 minutes
readonly MAX_STAGING_DURATION=900       # 15 minutes
readonly ENVIRONMENT="${ENVIRONMENT:-production}"

# Set duration limits based on environment
if [[ "$ENVIRONMENT" == "production" ]]; then
    MAX_DURATION=$MAX_PRODUCTION_DURATION
else
    MAX_DURATION=$MAX_STAGING_DURATION
fi

# Run with automatic termination
timeout "$MAX_DURATION" sudo memleak \
    -p "$PID" \
    -s "$SAMPLE_RATE" \
    -z "$MIN_SIZE" \
    --combined-only \
    > "$OUTPUT_FILE" 2>&1 || {
    echo "Sampling terminated after ${MAX_DURATION}s limit"
}

Monitoring & Alerting

When to Trigger Sampling

# monitoring-triggers.yaml
memory_leak_triggers:
  
  # Threshold-based triggers
  rss_growth_threshold: 100MB    # RSS growth in 1 hour
  memory_growth_rate: 50MB/hour  # Sustained growth rate
  oom_risk_score: 0.7           # Predictive OOM risk
  
  # Pattern-based triggers
  allocation_rate_spike: 5x      # 5x increase in allocation rate
  fragmentation_increase: 30%    # Memory fragmentation increase
  
  # Time-based triggers
  scheduled_health_check: "0 2 * * 0"  # Weekly at 2 AM Sunday
  
  # Manual triggers
  incident_response: true        # On-demand for incidents
  pre_deployment_check: true     # Before major deployments

Duration Recommendations

Environment Max Duration Sample Rate Trigger Frequency
Production 5 minutes 1000-2000 Incident-only
Staging 15 minutes 500-1000 Daily/Weekly
Development 30 minutes 100-500 On-demand
CI/CD 10 minutes 200-500 Per build

Overhead Monitoring

# overhead_monitor.py

class OverheadMonitor:
    def __init__(self):
        self.baseline_metrics = {}
        self.tracing_metrics = {}
    
    def capture_baseline(self, pid: int):
        """Capture performance metrics before tracing."""
        self.baseline_metrics = {
            'cpu_usage': self._get_cpu_usage(pid),
            'memory_usage': self._get_memory_usage(pid),
            'response_time': self._get_response_time(pid)
        }
    
    def monitor_overhead(self, pid: int) -> Dict:
        """Monitor overhead during tracing."""
        current_metrics = {
            'cpu_usage': self._get_cpu_usage(pid),
            'memory_usage': self._get_memory_usage(pid),
            'response_time': self._get_response_time(pid)
        }
        
        overhead = {}
        for metric, current_value in current_metrics.items():
            baseline_value = self.baseline_metrics.get(metric, 0)
            if baseline_value > 0:
                overhead[f'{metric}_overhead_percent'] = \
                    ((current_value - baseline_value) / baseline_value) * 100
        
        return overhead
    
    def should_terminate_early(self, overhead: Dict) -> bool:
        """Determine if tracing should be terminated due to high overhead."""
        
        # Terminate if CPU overhead exceeds 50%
        if overhead.get('cpu_usage_overhead_percent', 0) > 50:
            return True
            
        # Terminate if response time degrades >100%
        if overhead.get('response_time_overhead_percent', 0) > 100:
            return True
            
        return False

Comparison with Full Tracing

Overhead Differences

Metric Full Tracing Sampled (1000:1) Sampled (500:1) Sampled (100:1)
CPU Overhead 30-400% 10-15% 15-25% 25-35%
Memory Overhead Very High Low Moderate Moderate-High
Allocation Latency +100-400% +5-15% +10-25% +20-40%
Detection Accuracy 100% 85-95% 90-98% 95-99%
Production Suitable No Yes Limited Development Only

Accuracy Trade-offs

Full Tracing Advantages:

  • Captures every allocation and deallocation
  • Perfect accuracy for leak detection
  • Complete allocation timeline
  • No statistical estimation required

Sampled Tracing Trade-offs:

  • May miss very small, short-lived leaks
  • Requires statistical interpretation of results
  • Less precise leak size estimates
  • Potential for sampling bias in certain allocation patterns

Use Case Selection Guidelines

Choose Full Tracing When:
✓ Development environment debugging
✓ Reproducing known leaks with minimal performance requirements
✓ Complete accuracy is essential
✓ Short-duration debugging sessions (<5 minutes)

Choose Sampled Tracing When:
✓ Production environment monitoring needed
✓ Long-duration analysis required (>5 minutes)
✓ System performance must be preserved
✓ Statistical leak detection is sufficient
✓ Continuous or periodic monitoring

Migration Strategy

# Start with conservative sampling
sudo memleak -p $PID -s 2000 -z 2048 -T 300

# If no leaks found, increase sensitivity
sudo memleak -p $PID -s 1000 -z 1024 -T 300

# If leaks detected, increase sampling for better accuracy
sudo memleak -p $PID -s 500 -z 1024 -T 180

# For final confirmation, brief full tracing (development only)
sudo memleak -p $PID -z 1024 -T 60  # No sampling, 1 minute max

Advanced Configuration

Custom Allocation Function Targeting

# Target specific memory allocators
sudo memleak -p $PID -s 1000 -O libtcmalloc.so.4    # TCMalloc
sudo memleak -p $PID -s 1000 -O libjemalloc.so.2    # jemalloc
sudo memleak -p $PID -s 1000 -O libmimalloc.so      # mimalloc

Kernel vs User-Space Sampling

# User-space only (default)
sudo memleak -p $PID -s 1000

# Kernel allocations only
sudo memleak -K -s 500

# Combined user and kernel (higher overhead)
sudo memleak -p $PID -K -s 2000 --combined-only

Size-Stratified Sampling

# Different sampling rates for different allocation sizes
# Large allocations (>64KB) - sample more aggressively
sudo memleak -p $PID -s 100 -z 65536 -T 300 > large_allocs.txt &

# Medium allocations (1KB-64KB) - moderate sampling
sudo memleak -p $PID -s 1000 -z 1024 -Z 65535 -T 300 > medium_allocs.txt &

# Small allocations (<1KB) - light sampling only in development
sudo memleak -p $PID -s 5000 -Z 1023 -T 180 > small_allocs.txt &

Best Practices Summary

Production Deployment Checklist

  • Use sampling rates ≥500 (≤0.2% of allocations)
  • Set minimum size filters ≥1KB for production
  • Limit duration to ≤5 minutes in production
  • Use --combined-only flag for better performance
  • Monitor system overhead during sampling
  • Have automatic termination safeguards
  • Schedule during low-traffic periods when possible

Performance Optimization

  • Size filtering to focus on significant allocations
  • Appropriate sampling rate for allocation frequency
  • Duration limits with timeout protection
  • Combined statistics mode for reduced overhead
  • Stack depth limits for faster collection

Alert Integration

  • Automated triggering based on memory growth patterns
  • Statistical adjustment for sampling rate in alerts
  • Integration with existing monitoring infrastructure
  • Escalation procedures for confirmed leaks
  • Documentation of sampling methodology for incident response

See Also

⚠️ **GitHub.com Fallback** ⚠️