USAGE TRACKING - nself-org/cli GitHub Wiki
Part of nself v0.9.0 - Sprint 13: Billing Integration & Usage Tracking
Complete guide to usage metering, tracking, aggregation, and reporting across all billable services.
- Overview
- Tracked Services
- High-Volume Write Optimization
- Usage Recording
- Usage Aggregation
- Usage Reporting
- Usage Alerts
- Export Functionality
- Statistics & Analytics
- Cleanup & Maintenance
- API Reference
- Examples
The usage tracking system provides:
- High-volume write optimization with batch processing
- Six billable service types (API, storage, bandwidth, compute, database, functions)
- Real-time and aggregated reporting (hourly, daily, monthly)
- Multiple export formats (CSV, JSON, XLSX)
- Automated alerts based on quota thresholds
- Comprehensive analytics with trends and peak detection
βββββββββββββββββββ
β Service Calls β (API requests, storage, bandwidth, etc.)
ββββββββββ¬βββββββββ
β
βΌ
βββββββββββββββββββββββββββ
β usage_track_*() β Service-specific tracking functions
ββββββββββ¬βββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββ
β Batch Queue β Optional batch processing for high volume
β (usage_batch_add) β
ββββββββββ¬βββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββ
β billing_usage_records β PostgreSQL table
ββββββββββ¬βββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββ
β Aggregation & Reports β Hourly, daily, monthly summaries
βββββββββββββββββββββββββββ
The system tracks usage for six distinct services:
- Unit: Requests
- Tracking: Per endpoint, method, status code
-
Metadata:
{"endpoint":"/users","method":"GET","status":200}
- Unit: GB-hours (gigabyte-hours)
- Tracking: Bytes stored over time
-
Metadata:
{"bytes":1073741824,"hours":24}
- Unit: GB (gigabytes)
- Tracking: Data transfer (egress/ingress)
-
Metadata:
{"bytes":524288000,"direction":"egress"}
- Unit: CPU-hours
- Tracking: Container/VM compute time
- Metadata: Custom (container ID, instance type, etc.)
- Unit: Queries
- Tracking: Query count and duration
-
Metadata:
{"type":"SELECT","duration_ms":45}
- Unit: Invocations
- Tracking: Serverless function calls
-
Metadata:
{"function":"sendEmail","duration_ms":120,"memory_mb":256}
For services generating thousands of events per second, the system provides batch processing.
# Set batch size (default: 100 records)
export USAGE_BATCH_SIZE=500
# Set batch timeout (default: 5 seconds)
export USAGE_BATCH_TIMEOUT=10Add a usage record to the batch queue (non-blocking).
usage_batch_add "$customer_id" "api" 1 '{"endpoint":"/users","method":"GET"}'Manually flush the batch queue to database (uses PostgreSQL COPY for maximum performance).
usage_batch_flushInsert multiple records in a single transaction.
usage_batch_insert \
"api:1:{\"endpoint\":\"/users\"}" \
"api:1:{\"endpoint\":\"/posts\"}" \
"database:1:{\"type\":\"SELECT\"}"- Individual INSERT: ~1,000 records/second
- Batch INSERT (100 records): ~10,000 records/second
- PostgreSQL COPY: ~50,000+ records/second
Each service has a dedicated tracking function with proper unit conversion.
usage_track_api_request "/api/users" "GET" 200
usage_track_api_request "/api/posts" "POST" 201# Track 5GB stored for 24 hours
bytes=5368709120 # 5GB
usage_track_storage "$bytes" 24# Track 500MB egress
bytes=524288000
usage_track_bandwidth "$bytes" "egress"# Track 3600 seconds (1 hour) of CPU time
cpu_seconds=3600
usage_track_compute "$cpu_seconds" '{"instance":"t3.medium"}'usage_track_database_query "SELECT" 45 # 45ms duration
usage_track_database_query "INSERT" 120usage_track_function "sendEmail" 120 256 # 120ms, 256MB memory
usage_track_function "processImage" 3500 512All service functions ultimately call billing_record_usage():
billing_record_usage "service_name" quantity "metadata_json"Aggregate usage data for faster reporting and analysis.
# Hourly aggregation
usage_aggregate hourly "$customer_id" "2026-01-01" "2026-01-31"
# Daily aggregation
usage_aggregate daily "$customer_id" "2026-01-01" "2026-01-31"
# Monthly aggregation
usage_aggregate monthly "$customer_id" "2026-01-01" "2026-12-31"service_name | date | event_count | total_quantity | total_cost | avg_quantity
-------------+------------+-------------+----------------+------------+--------------
api | 2026-01-30 | 15420 | 15420 | 1.542 | 1.0
storage | 2026-01-30 | 24 | 120.5 | 0.1205 | 5.02
bandwidth | 2026-01-30 | 8934 | 45.67 | 2.2835 | 0.0051
The system uses a materialized view for even faster aggregations:
usage_refresh_summaryThis refreshes the billing_usage_daily_summary materialized view.
# Table format (default)
usage_get_all "2026-01-01" "2026-01-31"
# JSON format
usage_get_all "2026-01-01" "2026-01-31" "json"
# CSV format
usage_get_all "2026-01-01" "2026-01-31" "csv"
# Detailed breakdown
usage_get_all "2026-01-01" "2026-01-31" "table" "true"# API usage for January
usage_get_service "api" "2026-01-01" "2026-01-31"
# Storage usage with detailed timeline
usage_get_service "storage" "2026-01-01" "2026-01-31" "table" "true"
# Bandwidth usage as JSON
usage_get_service "bandwidth" "2026-01-01" "2026-01-31" "json"ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β USAGE SUMMARY β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ£
β Period: 2026-01-01 to 2026-01-31 β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ£
β Service β Usage β Unit β Cost β
β βββββββββββββββββββͺβββββββββββββββββͺβββββββββββββͺββββββββββββββ£
β api β 1.25M β requests β $125.00 β
β storage β 45.2 β GB-hours β $4.52 β
β bandwidth β 156.8 β GB β $7.84 β
β compute β 12.5 β CPU-hours β $0.63 β
β database β 45.6K β queries β $0.46 β
β functions β 8.9K β invocationsβ $1.78 β
ββββββββββββββββββββ§βββββββββββββββββ§βββββββββββββ§ββββββββββββββ
Automated monitoring of usage against quotas with three alert levels.
# Configure thresholds (percentage of quota)
export USAGE_ALERT_WARNING=75 # Default: 75%
export USAGE_ALERT_CRITICAL=90 # Default: 90%
export USAGE_ALERT_EXCEEDED=100 # Default: 100%# Check all services
usage_check_alerts
# Check specific service
usage_check_service_alert "$customer_id" "api"WARNING: api usage at 78% - 78000/100000
CRITICAL: storage usage at 92% - 9.2GB/10GB
QUOTA EXCEEDED: bandwidth - 105GB/100GB (105%)
# Last 7 days (default)
usage_get_alerts
# Last 30 days
usage_get_alerts 30[2026-01-30 14:23:45] warning ALERT for api: 78000/100000 (78%)
[2026-01-30 15:45:12] critical ALERT for storage: 9.2/10 (92%)
[2026-01-30 16:30:00] exceeded ALERT for bandwidth: 105/100 (105%)
Export usage data in multiple formats with automatic file naming.
# Export as CSV (default)
usage_export csv
# Export as JSON
usage_export json
# Export to specific file
usage_export csv "/path/to/usage.csv"
# Export with date range
usage_export csv "" "2026-01-01" "2026-01-31"
# Export specific service
usage_export json "" "2026-01-01" "2026-01-31" "api"Features:
- Headers included
- Comma-delimited
- Metadata as JSON string
- Sorted by timestamp (newest first)
Format:
timestamp,service,quantity,unit_cost,total_cost,metadata
2026-01-30 16:45:23,api,1,0.0001,0.0001,"{""endpoint"":""/users"",""method"":""GET""}"
2026-01-30 16:45:22,bandwidth,0.000512,0.01,0.00000512,"{""bytes"":524288,""direction"":""egress""}"Features:
- Complete metadata preserved
- Summary statistics included
- Nested structure for clarity
Format:
{
"customer_id": "cus_abc123",
"export_date": "2026-01-30T16:45:23Z",
"period": {
"start": "2026-01-01",
"end": "2026-01-31"
},
"usage_records": [
{
"timestamp": "2026-01-30T16:45:23Z",
"service": "api",
"quantity": 1,
"unit_cost": 0.0001,
"total_cost": 0.0001,
"metadata": {"endpoint": "/users", "method": "GET"}
}
],
"summary": {
"total_events": 125000,
"total_cost": 140.23,
"services": ["api", "storage", "bandwidth", "compute", "database", "functions"]
}
}Requires csvkit or similar tools. Falls back to CSV if not available.
# Install csvkit (if needed)
pip install csvkit
# Export as XLSX
usage_export xlsxAdvanced analytics functions for usage patterns and trends.
Get comprehensive stats for a service:
# API statistics for last 30 days
usage_get_stats "api" 30
# Storage statistics for last 7 days
usage_get_stats "storage" 7Output:
service_name | total_events | total_quantity | avg_quantity | p95_quantity | p99_quantity
-------------+--------------+----------------+--------------+--------------+--------------
api | 125000 | 125000 | 1.0 | 1.0 | 1.0
Day-over-day comparison with percentage change:
# All services, last 7 days
usage_get_trends
# Specific service, last 30 days
usage_get_trends "api" 30Output:
date | service_name | daily_total | previous_day | percent_change
-----------+--------------+-------------+--------------+----------------
2026-01-30 | api | 5420 | 5123 | 5.80
2026-01-29 | api | 5123 | 5456 | -6.10
Find highest usage periods:
# Top 10 hours for API
usage_get_peaks "api" "hourly" 10
# Top 5 days for storage
usage_get_peaks "storage" "daily" 5Output:
period | total_usage | event_count
--------------------+-------------+-------------
2026-01-30 14:00:00 | 1250 | 1250
2026-01-30 15:00:00 | 1180 | 1180
Archive and delete usage records older than N days:
# Archive records older than 90 days (default)
usage_archive
# Archive records older than 30 days
usage_archive 30
# Archive to specific file
usage_archive 90 "/path/to/archive.csv"Process:
- Export records older than cutoff to CSV
- Delete archived records from database
- Report number of archived records
Flush pending batches and clean temporary files:
usage_cleanup_batchRecommended:
-
Daily: Refresh materialized view (
usage_refresh_summary) -
Weekly: Check alerts (
usage_check_alerts) -
Monthly: Archive old records (
usage_archive) - Quarterly: Export historical data for long-term storage
| Variable | Default | Description |
|---|---|---|
USAGE_BATCH_SIZE |
100 | Records per batch flush |
USAGE_BATCH_TIMEOUT |
5 | Seconds before auto-flush |
USAGE_ALERT_WARNING |
75 | Warning threshold (%) |
USAGE_ALERT_CRITICAL |
90 | Critical threshold (%) |
USAGE_ALERT_EXCEEDED |
100 | Exceeded threshold (%) |
| Function | Parameters | Description |
|---|---|---|
usage_get_all |
start, end, format, detailed | Get all usage for period |
usage_get_service |
service, start, end, format, detailed | Get service usage |
usage_export |
format, file, start, end, service | Export usage data |
usage_aggregate |
period, customer_id, start, end | Aggregate usage |
usage_check_alerts |
- | Check all quota alerts |
usage_get_stats |
service, days | Get usage statistics |
usage_get_trends |
service, days | Get usage trends |
usage_archive |
days_to_keep, archive_file | Archive old records |
| Function | Parameters | Description |
|---|---|---|
usage_track_api_request |
endpoint, method, status | Track API request |
usage_track_storage |
bytes, hours | Track storage usage |
usage_track_bandwidth |
bytes, direction | Track bandwidth |
usage_track_compute |
cpu_seconds, metadata | Track compute time |
usage_track_database_query |
type, duration_ms | Track DB query |
usage_track_function |
name, duration_ms, memory_mb | Track function |
| Function | Parameters | Description |
|---|---|---|
usage_init_batch |
- | Initialize batch processing |
usage_batch_add |
customer_id, service, qty, metadata | Add to batch |
usage_batch_flush |
- | Flush batch to database |
usage_batch_insert |
records... | Bulk insert records |
#!/usr/bin/env bash
source "src/lib/billing/usage.sh"
# Track various API endpoints
usage_track_api_request "/api/users" "GET" 200
usage_track_api_request "/api/users/123" "GET" 200
usage_track_api_request "/api/posts" "POST" 201
usage_track_api_request "/api/invalid" "GET" 404
# View usage
usage_get_service "api" "$(date -u +"%Y-%m-%d 00:00:00")" "$(date -u +"%Y-%m-%d 23:59:59")"#!/usr/bin/env bash
source "src/lib/billing/usage.sh"
# Initialize batch processing
usage_init_batch
# Get customer ID
customer_id=$(billing_get_customer_id)
# Track 1000 API requests (batched)
for i in {1..1000}; do
usage_batch_add "$customer_id" "api" 1 "{\"endpoint\":\"/test\",\"i\":$i}"
done
# Batch auto-flushes at 100 records, but flush any remaining
usage_batch_flush#!/usr/bin/env bash
source "src/lib/billing/usage.sh"
# Calculate storage usage
total_bytes=$(du -sb /data | cut -f1)
hours_since_last=$(( ($(date +%s) - $(stat -f %m /data/.last_check)) / 3600 ))
# Track usage
usage_track_storage "$total_bytes" "$hours_since_last"
# Update last check
touch /data/.last_check
# Check if approaching quota
usage_check_service_alert "$(billing_get_customer_id)" "storage"#!/usr/bin/env bash
source "src/lib/billing/usage.sh"
# Get today's date range
today=$(date -u +"%Y-%m-%d")
start="${today} 00:00:00"
end="${today} 23:59:59"
# Generate report
printf "Daily Usage Report - %s\n\n" "$today"
usage_get_all "$start" "$end" "table" "true"
# Export to file
output_file="/var/reports/usage_${today}.csv"
usage_export csv "$output_file" "$start" "$end"
printf "\nExported to: %s\n" "$output_file"#!/usr/bin/env bash
source "src/lib/billing/usage.sh"
# Analyze API usage trends
printf "API Usage Trends (Last 7 Days)\n"
printf "================================\n\n"
usage_get_trends "api" 7
# Find peak usage hours
printf "\nTop 5 Peak Hours\n"
printf "================\n\n"
usage_get_peaks "api" "hourly" 5
# Get detailed statistics
printf "\nDetailed Statistics (Last 30 Days)\n"
printf "==================================\n\n"
usage_get_stats "api" 30#!/usr/bin/env bash
source "src/lib/billing/usage.sh"
# Check quotas for all services
printf "Checking usage quotas...\n\n"
usage_check_alerts 2>&1 | while IFS= read -r line; do
if [[ "$line" =~ "EXCEEDED" ]]; then
# Send urgent alert
echo "$line" | mail -s "URGENT: Quota Exceeded" [email protected]
elif [[ "$line" =~ "CRITICAL" ]]; then
# Send warning
echo "$line" | mail -s "WARNING: Quota Critical" [email protected]
fi
# Log all alerts
printf "%s\n" "$line"
done
# Show recent alert history
printf "\nRecent Alerts (Last 7 Days)\n"
printf "===========================\n\n"
usage_get_alerts 7#!/usr/bin/env bash
source "src/lib/billing/usage.sh"
# Get last month's date range
if date -v-1m >/dev/null 2>&1; then
# macOS
start=$(date -v-1m -v1d +"%Y-%m-01 00:00:00")
end=$(date -v-1m -v+1m -v-1d +"%Y-%m-%d 23:59:59")
else
# Linux
start=$(date -d "last month" +"%Y-%m-01 00:00:00")
end=$(date -d "$(date +%Y-%m-01) -1 day" +"%Y-%m-%d 23:59:59")
fi
month=$(date -d "$start" +"%Y-%m" 2>/dev/null || date -j -f "%Y-%m-%d" "$(echo $start | cut -d' ' -f1)" +"%Y-%m")
printf "Generating monthly usage report for %s...\n" "$month"
# Export in multiple formats
usage_export csv "/var/billing/usage_${month}.csv" "$start" "$end"
usage_export json "/var/billing/usage_${month}.json" "$start" "$end"
# Archive old records (keep last 90 days)
usage_archive 90 "/var/billing/archive/usage_${month}_archive.csv"
printf "\nMonthly export complete\n"For systems generating >10,000 events/second:
-
Enable batch processing:
export USAGE_BATCH_SIZE=1000 -
Use PostgreSQL COPY via
usage_batch_flush() -
Partition the table by date:
CREATE TABLE billing_usage_records_2026_01 PARTITION OF billing_usage_records FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');
-
Use async workers to process batches
-
Indexes (already created):
(customer_id, service_name)(customer_id, recorded_at)-
(aggregated)(partial index for unaggregated records)
-
Materialized view for daily summaries
-
Regular VACUUM and ANALYZE:
VACUUM ANALYZE billing_usage_records;
- Archive old records monthly
- Compress archives (gzip recommended)
- Use object storage (S3, MinIO) for long-term archives
- Billing & Usage
- Quota Management
- Stripe Integration
- Database Schema
Version: 0.9.0 Last Updated: 2026-01-30 Status: Production Ready