V1.7.7_MQTT_IMPROVEMENTS - Alteriom/painlessMesh GitHub Wiki

Version 1.7.7 - Enhanced MQTT Communication for Metrics & Health Monitoring

Overview

Version 1.7.7 introduces two new specialized packages designed to enhance MQTT communication efficiency for monitoring mesh networks. These packages provide comprehensive visibility into system performance, proactive problem detection, and predictive maintenance capabilities.

New Package Types

MetricsPackage (Type 204)

A comprehensive performance metrics package designed for real-time dashboards, capacity planning, and network optimization.

Purpose

  • Real-time Monitoring - Track system performance in real-time
  • Capacity Planning - Identify bottlenecks and plan upgrades
  • Performance Optimization - Analyze and improve system efficiency
  • Troubleshooting - Diagnose performance issues quickly

Key Metrics Categories

CPU and Processing:

  • cpuUsage (uint8_t) - CPU usage percentage (0-100)
  • loopIterations (uint32_t) - Loop iterations per second
  • taskQueueSize (uint16_t) - Number of pending tasks

Memory Metrics:

  • freeHeap (uint32_t) - Current free heap memory in bytes
  • minFreeHeap (uint32_t) - Minimum free heap since boot
  • heapFragmentation (uint32_t) - Heap fragmentation percentage
  • maxAllocHeap (uint32_t) - Largest allocatable block

Network Performance:

  • bytesReceived (uint32_t) - Total bytes received
  • bytesSent (uint32_t) - Total bytes sent
  • packetsReceived (uint16_t) - Total packets received
  • packetsSent (uint16_t) - Total packets sent
  • packetsDropped (uint16_t) - Packets dropped
  • currentThroughput (uint16_t) - Current throughput in bytes/sec

Timing and Latency:

  • avgResponseTime (uint32_t) - Average response time in microseconds
  • maxResponseTime (uint32_t) - Maximum response time in microseconds
  • avgMeshLatency (uint16_t) - Average mesh latency in milliseconds

Connection Quality:

  • connectionQuality (uint8_t) - Overall connection quality score (0-100)
  • wifiRSSI (int8_t) - WiFi RSSI in dBm

Metadata:

  • collectionTimestamp (uint32_t) - When metrics were collected
  • collectionInterval (uint32_t) - Interval between collections in ms

Example Usage

#include "alteriom_sensor_package.hpp"
using namespace alteriom;

void collectAndSendMetrics() {
  MetricsPackage metrics;
  metrics.from = mesh.getNodeId();
  
  // Collect CPU metrics
  metrics.cpuUsage = calculateCPUUsage();
  metrics.loopIterations = getLoopsPerSecond();
  metrics.taskQueueSize = scheduler.size();
  
  // Collect memory metrics
  metrics.freeHeap = ESP.getFreeHeap();
  metrics.minFreeHeap = getMinHeapSinceBoot();
  #ifdef ESP32
  metrics.maxAllocHeap = ESP.getMaxAllocHeap();
  #else
  metrics.heapFragmentation = ESP.getHeapFragmentation();
  metrics.maxAllocHeap = ESP.getMaxFreeBlockSize();
  #endif
  
  // Collect network metrics
  metrics.bytesReceived = getTotalBytesRx();
  metrics.bytesSent = getTotalBytesTx();
  metrics.currentThroughput = calculateThroughput();
  
  // Connection quality
  metrics.connectionQuality = calculateConnectionQuality();
  metrics.wifiRSSI = WiFi.RSSI();
  
  // Metadata
  metrics.collectionTimestamp = mesh.getNodeTime();
  metrics.collectionInterval = 30000;  // 30 seconds
  
  // Send metrics
  mesh.sendBroadcast(metrics.toJsonString());
}

MQTT Integration

When using with the MQTT bridge, metrics are published to:

  • mesh/metrics/{node_id} - Per-node metrics
  • mesh/metrics/aggregated - Aggregated mesh-wide metrics

Dashboard Integration

Grafana:

{
  "cpu_usage": "${cpuUsage}",
  "free_heap": "${freeHeap}",
  "throughput": "${currentThroughput}",
  "rssi": "${wifiRSSI}",
  "quality": "${connectionQuality}"
}

InfluxDB Line Protocol:

metrics,node_id=12345 cpu=${cpuUsage},heap=${freeHeap},throughput=${currentThroughput} ${timestamp}

HealthCheckPackage (Type 605)

A proactive health monitoring package designed for early problem detection, predictive maintenance, and automated alerting.

Purpose

  • Proactive Detection - Identify problems before they cause failures
  • Predictive Maintenance - Estimate time to failure and plan maintenance
  • Automated Alerting - Trigger alerts based on health thresholds
  • Root Cause Analysis - Provide detailed health indicators for troubleshooting

Health Status Levels

  • 2 (Healthy) - All systems operating normally
  • 1 (Warning) - Some issues detected, monitoring required
  • 0 (Critical) - Immediate attention required

Problem Flags (Bit Flags)

Flag Value Description
Low Memory 0x0001 Free memory below threshold
High CPU 0x0002 CPU usage above threshold
Connection Instability 0x0004 Frequent reconnections
High Packet Loss 0x0008 Packet loss above threshold
Network Congestion 0x0010 Network congestion detected
Low Battery 0x0020 Battery level critical
Thermal Warning 0x0040 Temperature above safe limit
Mesh Partition 0x0080 Mesh network partitioned
OTA in Progress 0x0100 Firmware update active
Configuration Error 0x0200 Configuration issue detected

Health Scores (0-100)

Each component (memory, network, performance) has a health score:

  • 90-100 - Excellent
  • 70-89 - Good
  • 50-69 - Fair
  • 30-49 - Poor
  • 0-29 - Critical

Key Fields

Overall Health:

  • healthStatus (uint8_t) - Overall status (0-2)
  • problemFlags (uint16_t) - Bit flags for specific problems

Component Health:

  • memoryHealth (uint8_t) - Memory health score (0-100)
  • memoryTrend (uint32_t) - Memory loss rate in bytes/hour
  • networkHealth (uint8_t) - Network health score (0-100)
  • packetLossPercent (uint8_t) - Current packet loss percentage
  • reconnectionCount (uint8_t) - Reconnections in last hour
  • performanceHealth (uint8_t) - Performance health score (0-100)
  • missedDeadlines (uint32_t) - Missed task deadlines
  • maxLoopTime (uint16_t) - Maximum loop execution time in ms

Environmental:

  • temperature (int8_t) - Device temperature in Celsius
  • temperatureHealth (uint8_t) - Temperature health score

Stability:

  • uptime (uint32_t) - Uptime in seconds
  • crashCount (uint16_t) - Crash/restart count
  • lastRebootReason (uint32_t) - Last reboot reason code

Predictive:

  • estimatedTimeToFailure (uint16_t) - Estimated hours until failure (0=unknown)
  • recommendations (TSTRING) - Recommended actions

Example Usage

void performHealthCheck() {
  HealthCheckPackage health;
  health.from = mesh.getNodeId();
  
  // Calculate component health scores
  uint8_t memHealth = calculateMemoryHealth();
  uint8_t netHealth = calculateNetworkHealth();
  uint8_t perfHealth = calculatePerformanceHealth();
  
  // Determine overall health status
  if (memHealth < 30 || netHealth < 30 || perfHealth < 30) {
    health.healthStatus = 0;  // Critical
  } else if (memHealth < 60 || netHealth < 60 || perfHealth < 60) {
    health.healthStatus = 1;  // Warning
  } else {
    health.healthStatus = 2;  // Healthy
  }
  
  // Set problem flags
  health.problemFlags = 0;
  if (memHealth < 60) health.problemFlags |= 0x0001;
  if (perfHealth < 60) health.problemFlags |= 0x0002;
  if (netHealth < 60) health.problemFlags |= 0x0004;
  
  // Component health
  health.memoryHealth = memHealth;
  health.memoryTrend = calculateMemoryTrend();
  health.networkHealth = netHealth;
  health.packetLossPercent = getPacketLossPercent();
  health.performanceHealth = perfHealth;
  
  // Predictive indicators
  if (health.memoryTrend > 0) {
    uint32_t freeHeap = ESP.getFreeHeap();
    health.estimatedTimeToFailure = freeHeap / health.memoryTrend;
  }
  
  // Recommendations
  if (health.healthStatus == 0) {
    health.recommendations = "CRITICAL: Immediate attention required";
  } else if (health.healthStatus == 1) {
    if (memHealth < 60) {
      health.recommendations = "Increase memory allocation";
    } else if (netHealth < 60) {
      health.recommendations = "Check network connections";
    }
  } else {
    health.recommendations = "System operating normally";
  }
  
  // Send health check
  mesh.sendBroadcast(health.toJsonString());
}

Alert Integration

Home Assistant:

- platform: mqtt
  name: "Mesh Node Health"
  state_topic: "mesh/health/12345"
  value_template: "{{ value_json.healthStatus }}"
  json_attributes_topic: "mesh/health/12345"
  json_attributes_template: "{{ value_json | tojson }}"

Alerting Rules:

alerts:
  - name: critical_health
    condition: healthStatus == 0
    action: send_notification
  - name: memory_warning
    condition: memoryHealth < 60
    action: log_warning
  - name: predicted_failure
    condition: estimatedTimeToFailure < 24
    action: schedule_maintenance

Collection Intervals

Recommended Intervals

Package Type Interval Use Case
MetricsPackage 30-60s Normal monitoring
MetricsPackage 10-30s Active troubleshooting
MetricsPackage 5-10s Critical performance analysis
HealthCheckPackage 60s Normal health monitoring
HealthCheckPackage 30s Warning state monitoring
HealthCheckPackage 10s Critical state monitoring

Configurable Collection

// In setup()
#define METRICS_INTERVAL 30000   // 30 seconds
#define HEALTH_INTERVAL 60000    // 60 seconds

Task taskMetrics(METRICS_INTERVAL, TASK_FOREVER, &sendMetrics);
Task taskHealth(HEALTH_INTERVAL, TASK_FOREVER, &sendHealthCheck);

userScheduler.addTask(taskMetrics);
userScheduler.addTask(taskHealth);

taskMetrics.enable();
taskHealth.enable();

// Dynamic adjustment based on health status
void adjustCollectionIntervals(uint8_t healthStatus) {
  if (healthStatus == 0) {  // Critical
    taskMetrics.setInterval(10000);   // 10 seconds
    taskHealth.setInterval(10000);
  } else if (healthStatus == 1) {  // Warning
    taskMetrics.setInterval(20000);   // 20 seconds
    taskHealth.setInterval(30000);
  } else {  // Healthy
    taskMetrics.setInterval(60000);   // 60 seconds
    taskHealth.setInterval(60000);
  }
}

MQTT Bridge Integration

Publishing Topics

Per-Node Topics

  • mesh/metrics/{node_id} - Individual node metrics
  • mesh/health/{node_id} - Individual node health checks

Aggregated Topics

  • mesh/metrics/aggregated - Mesh-wide aggregated metrics
  • mesh/health/summary - Overall mesh health summary
  • mesh/alerts - Active alerts across the mesh

Message Format (Updated for @alteriom/mqtt-schema v0.7.2+)

Packages now include the message_type field for 90% faster message classification:

{
  "schema_version": 1,
  "device_id": "ALT-12345",
  "timestamp": "2025-10-23T21:30:00Z",
  "type": 204,
  "message_type": 204,
  "from": 12345,
  "cpuUsage": 45,
  "freeHeap": 100000,
  "currentThroughput": 8192,
  "connectionQuality": 85,
  "wifiRSSI": -55,
  ...
}

Message Type Codes (mqtt-schema v0.7.2+):

For performance optimization and standardized routing, all Alteriom packages include the message_type field:

Code Constant Message Type Category Description PainlessMesh Package
200 SENSOR_DATA sensor_data telemetry Sensor telemetry readings SensorPackage
202 SENSOR_STATUS sensor_status telemetry Sensor status change StatusPackage
204 SENSOR_METRICS sensor_metrics telemetry Sensor health and performance metrics MetricsPackage
400 COMMAND command control Device control command CommandPackage
600 MESH_NODE_LIST mesh_node_list mesh Mesh node inventory MeshNodeListPackage
601 MESH_TOPOLOGY mesh_topology mesh Mesh network topology MeshTopologyPackage
602 MESH_ALERT mesh_alert mesh Mesh network alert MeshAlertPackage
603 MESH_BRIDGE mesh_bridge mesh Mesh protocol bridge MeshBridgePackage
604 MESH_STATUS mesh_status mesh Mesh network health status EnhancedStatusPackage
605 MESH_METRICS mesh_metrics mesh Mesh network performance metrics HealthCheckPackage

Key Points:

  • The message_type field enables 90% faster message classification compared to parsing JSON
  • All Alteriom packages align with @alteriom/mqtt-schema v0.7.2+ standards
  • Message type codes are consistent across MQTT bridge, gateway, and mesh nodes

Implementation Guide

Basic Implementation

See examples/alteriom/metrics_health_node.ino for a complete working example.

Integration with Existing Code

#include "alteriom_sensor_package.hpp"
using namespace alteriom;

// Add to your existing mesh node
void setup() {
  // ... existing mesh setup ...
  
  // Add metrics collection task
  userScheduler.addTask(Task(30000, TASK_FOREVER, []() {
    MetricsPackage metrics;
    // ... populate metrics ...
    mesh.sendBroadcast(metrics.toJsonString());
  }));
  
  // Add health check task
  userScheduler.addTask(Task(60000, TASK_FOREVER, []() {
    HealthCheckPackage health;
    // ... populate health data ...
    mesh.sendBroadcast(health.toJsonString());
  }));
}

Gateway Bridge Integration

// In MQTT bridge receivedCallback
void receivedCallback(uint32_t from, String& msg) {
  DynamicJsonDocument doc(2048);
  deserializeJson(doc, msg);
  
  uint8_t msgType = doc["type"];
  
  switch(msgType) {
    case 204:  // MetricsPackage
      publishMetricsToMQTT(from, doc);
      break;
    case 605:  // HealthCheckPackage (MESH_METRICS)
      publishHealthToMQTT(from, doc);
      checkForAlerts(doc);
      break;
  }
}

void publishMetricsToMQTT(uint32_t from, JsonDocument& doc) {
  String topic = "mesh/metrics/" + String(from);
  String payload;
  serializeJson(doc, payload);
  mqttClient.publish(topic.c_str(), payload.c_str());
}

void checkForAlerts(JsonDocument& doc) {
  uint8_t healthStatus = doc["health"];
  uint16_t problemFlags = doc["problems"];
  
  if (healthStatus == 0) {
    String alertTopic = "mesh/alerts/critical";
    String alertMsg = "Critical health on node " + String(doc["from"].as<uint32_t>());
    mqttClient.publish(alertTopic.c_str(), alertMsg.c_str());
  }
}

Performance Considerations

Memory Usage

  • MetricsPackage: ~200 bytes per message
  • HealthCheckPackage: ~250 bytes per message (including recommendations string)
  • Total overhead: <1KB for both packages with reasonable collection intervals

Network Bandwidth

With 10 nodes and recommended intervals:

  • Metrics (30s): 10 nodes × 200 bytes / 30s = ~67 bytes/sec
  • Health (60s): 10 nodes × 250 bytes / 60s = ~42 bytes/sec
  • Total: ~109 bytes/sec = minimal overhead

Optimization Tips

  1. Adjust intervals based on load - Increase intervals during normal operation
  2. Use selective reporting - Only send metrics that changed significantly
  3. Aggregate at gateway - Combine multiple node metrics before MQTT publish
  4. Compress recommendations - Use short, standardized recommendation codes
  5. Throttle during congestion - Reduce collection frequency when network is busy

Testing

Unit Tests

Run the comprehensive test suite:

./bin/catch_metrics_health_packages

Tests validate:

  • Serialization/deserialization
  • Field preservation
  • Edge cases (min/max values)
  • Problem flag handling
  • Health status levels
  • Integration with painlessMesh plugin system

Integration Testing

// Test metrics collection
void testMetrics() {
  MetricsPackage metrics;
  metrics.from = 12345;
  metrics.cpuUsage = 45;
  metrics.freeHeap = 100000;
  
  auto var = protocol::Variant(&metrics);
  auto metrics2 = var.to<MetricsPackage>();
  
  assert(metrics2.cpuUsage == 45);
  assert(metrics2.freeHeap == 100000);
}

// Test health monitoring
void testHealth() {
  HealthCheckPackage health;
  health.from = 12345;
  health.healthStatus = 1;
  health.problemFlags = 0x0001;
  
  auto var = protocol::Variant(&health);
  auto health2 = var.to<HealthCheckPackage>();
  
  assert(health2.healthStatus == 1);
  assert(health2.problemFlags == 0x0001);
}

Migration from Previous Versions

From v1.7.6

No breaking changes. Simply add the new packages to your code:

// Add new includes
#include "alteriom_sensor_package.hpp"
using namespace alteriom;

// Add new collection tasks
// ... see examples above ...

Backward Compatibility

  • All existing packages (200-203) continue to work unchanged
  • New packages (204, 604, 605) are optional additions
  • No changes required to existing code
  • Can be adopted incrementally

Best Practices

  1. Start with conservative intervals - Begin with 60s intervals and adjust based on needs
  2. Monitor memory usage - Watch heap fragmentation when enabling new packages
  3. Implement health-based throttling - Reduce collection frequency when unhealthy
  4. Use problem flags effectively - Check specific flags rather than just status
  5. Act on recommendations - Implement automated responses to common recommendations
  6. Set up alerting - Configure alerts for critical health status
  7. Track trends - Use memoryTrend to detect slow memory leaks
  8. Test thoroughly - Validate metrics accuracy in your specific environment

Troubleshooting

High Memory Usage

  • Increase collection intervals
  • Reduce number of active packages
  • Use shorter recommendation strings
  • Check for memory leaks using memoryTrend

Network Congestion

  • Increase collection intervals
  • Use SINGLE routing instead of BROADCAST for some packages
  • Implement throttling based on network health
  • Reduce message size by removing optional fields

Inaccurate Metrics

  • Calibrate CPU usage calculation for your use case
  • Validate timing measurements
  • Check clock synchronization across mesh
  • Verify RSSI readings match reality

Future Enhancements

Planned for v1.8.0:

  • Compressed metric packages for large meshes
  • Historical trend storage in gateway
  • Automatic threshold tuning
  • Machine learning-based failure prediction
  • Integration with cloud monitoring services

Enhanced MQTT Bridge

Overview

The enhanced MQTT bridge (examples/bridge/enhanced_mqtt_bridge.hpp) extends the basic MQTT status bridge with command handlers and aggregation capabilities.

Features

  1. Command Handlers - Request metrics and health checks on-demand
  2. Aggregated Statistics - Mesh-wide metrics and health summaries
  3. Automatic Caching - Stores recent metrics for aggregation
  4. Alert Detection - Automatic critical health alerting
  5. Response Topics - Dedicated topics for command responses

MQTT Topics

Subscribe Topics (Commands)

Topic Payload Description
mesh/command/request_metrics {"node_id": 0} Request metrics (0=all nodes)
mesh/command/request_health {"node_id": 12345} Request health check
mesh/command/get_aggregated {} Get current aggregated stats

Publish Topics (Data)

Topic Description Update Frequency
mesh/metrics/{node_id} Individual node metrics When received from node
mesh/health/{node_id} Individual node health When received from node
mesh/aggregated/metrics Mesh-wide metrics Configurable (default 60s)
mesh/aggregated/health Mesh health summary Configurable (default 60s)
mesh/alerts/critical Critical health alerts When critical detected
mesh/response/metrics Metrics command response When node responds
mesh/response/health Health command response When node responds

Aggregated Metrics Format

{
  "node_count": 10,
  "avg_cpu": 42,
  "avg_heap": 125000,
  "min_heap": 95000,
  "max_heap": 150000,
  "total_throughput": 81920,
  "avg_quality": 87,
  "min_quality": 65,
  "max_quality": 98,
  "avg_rssi": -58,
  "timestamp": "2025-10-23T21:30:00Z"
}

Aggregated Health Format

{
  "node_count": 10,
  "mesh_health": 2,
  "healthy_nodes": 8,
  "warning_nodes": 2,
  "critical_nodes": 0,
  "aggregated_problems": 5,
  "avg_memory_health": 82,
  "avg_network_health": 88,
  "avg_performance_health": 90,
  "timestamp": "2025-10-23T21:30:00Z"
}

Example Usage

Request Metrics from All Nodes

mosquitto_pub -h localhost -t mesh/command/request_metrics \
  -m '{"node_id": 0}'

Request Health from Specific Node

mosquitto_pub -h localhost -t mesh/command/request_health \
  -m '{"node_id": 12345}'

Get Current Aggregated Statistics

mosquitto_pub -h localhost -t mesh/command/get_aggregated -m '{}'

Subscribe to Aggregated Metrics

mosquitto_sub -h localhost -t mesh/aggregated/metrics

Subscribe to Critical Alerts

mosquitto_sub -h localhost -t mesh/alerts/critical

Integration Example

#include "enhanced_mqtt_bridge.hpp"

// In setup()
EnhancedMqttBridge bridge(mesh, mqttClient);
bridge.setTopicPrefix("mesh/");
bridge.enableAggregation(true);
bridge.setAggregationInterval(60000);  // 60 seconds
bridge.begin();

// In loop()
bridge.update();

// In MQTT callback
void mqttCallback(char* topic, byte* payload, unsigned int length) {
  String topicStr = String(topic);
  String payloadStr = String((char*)payload).substring(0, length);
  bridge.handleMQTTMessage(topicStr, payloadStr);
}

Configuration

// Set custom topic prefix
bridge.setTopicPrefix("alteriom/mesh/");

// Set device ID
bridge.setDeviceId("GATEWAY-001");

// Enable/disable aggregation
bridge.enableAggregation(true);

// Set aggregation interval
bridge.setAggregationInterval(30000);  // 30 seconds

Mesh Node Implementation

Nodes must respond to command requests:

void receivedCallback(uint32_t from, String& msg) {
  DynamicJsonDocument doc(1024);
  deserializeJson(doc, msg);
  
  uint8_t msgType = doc["type"];
  
  if (msgType == 201) {  // CommandPackage
    uint8_t command = doc["cmd"];
    
    if (command == 210) {  // Request metrics
      sendMetrics();
    } else if (command == 211) {  // Request health
      sendHealthCheck();
    }
  }
}

Performance Considerations

Memory Usage:

  • Cache size: ~100 bytes per node (max 20 nodes)
  • Total overhead: ~2KB for bridge + caching

Network Bandwidth:

  • Aggregated metrics: ~200 bytes every 60s
  • Per-node metrics: ~200 bytes when received
  • Critical alerts: ~150 bytes when detected

Optimization:

  • Adjust cache size via MAX_STORED_NODES constant
  • Increase aggregation interval for lower bandwidth
  • Disable aggregation if not needed

Support

Changelog

v1.7.7 (2025-10-23)

Added:

  • MetricsPackage (Type 204) for comprehensive performance monitoring
  • HealthCheckPackage (Type 605) for proactive health monitoring (MESH_METRICS)
  • Complete test suite for new packages
  • Example implementation in metrics_health_node.ino
  • Documentation for MQTT integration
  • Dashboard integration examples

Improved:

  • MQTT communication efficiency for metrics and health monitoring
  • Problem detection and alerting capabilities
  • Predictive maintenance support
  • Memory leak detection

Compatibility:

  • 100% backward compatible with v1.7.6
  • All existing packages (200-203) unchanged
  • Optional adoption of new features
⚠️ **GitHub.com Fallback** ⚠️