Cognitive Ergonomics Guide: Training Pipeline Mental Model

Purpose: This document provides visual and conceptual guidance for understanding the q_mini_wasm_v2 training pipeline. It follows cognitive ergonomics principles to reduce mental load when working with the system.

Quick Mental Model: The "Knowledge Refinery"

Think of the training pipeline as a knowledge refinery that transforms raw information into trained expertise:

┌─────────────────────────────────────────────────────────────────────────────┐
│                          KNOWLEDGE REFINERY                                  │
│                                                                              │
│  Raw Knowledge Sources                                                       │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                                   │
│  │ Wolfram  │  │ PubChem  │  │  OEIS    │  ← External APIs                 │
│  │ (Math)   │  │(Chemistry│  │(Sequences│                                   │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘                                   │
│       │             │             │                                          │
│       └─────────────┴─────────────┘                                          │
│                     │                                                        │
  // ... (truncated)
  // See source for complete code

Decision Flow: "Should I Trigger Betti Guidance?"

                    ┌─────────────────┐
                    │ Training Loop   │
                    │ Batch Complete  │
                    └────────┬────────┘
                             │
                             ▼
              ┌──────────────────────────────┐
              │ batch_count % interval == 0 ?  │
              └────────┬──────────┬──────────┘
                       │          │
                    YES │          │ NO
  // ... (truncated)
  // See source for complete code

Data Flow: From API to Trained Expert

┌──────────────────────────────────────────────────────────────────────────────┐
│ STAGE 1: ACQUISITION                                                         │
├──────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  HTTP Request                    Response Parser                             │
│  ┌─────────┐                   ┌───────────────┐                            │
│  │ GET api │ ───200 OK─────▶   │ JSON → float │  ← Error handling          │
│  │ /query  │    [1.2, 3.4]     │ vector        │    → exponential backoff   │
│  └─────────┘                   └───────┬───────┘                            │
│                                          │                                   │
│  Rate Limit: 100 req/min                 ▼                                   │
  // ... (truncated)
  // See source for complete code

Configuration Decision Tree

┌─────────────────────────────────────────────────────────────┐
│                START: Configure Pipeline                    │
└───────────────────────┬─────────────────────────────────────┘
                        │
                        ▼
            ┌───────────────────────┐
            │ Need real-time API    │
            │ data?                 │
            └───────────┬───────────┘
                        │
            ┌───────────┴───────────┐
  // ... (truncated)
  // See source for complete code

Error Handling Mental Model

┌────────────────────────────────────────────────────────────────┐
│                    ERROR LAYERS                                  │
├────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Layer 4: API Errors (recoverable)                              │
│  ┌────────────────────────────────────────────────────────┐   │
│  │ Rate limit exceeded → Backoff → Retry                  │   │
│  │ Network timeout → Log → Use synthetic fallback         │   │
│  │ Invalid response → Skip batch → Continue               │   │
│  └────────────────────────────────────────────────────────┘   │
│                          │                                      │
  // ... (truncated)
  // See source for complete code

Metric Interpretation Guide

Metric	Healthy Range	Action if Outside
`ff_goodness_delta`	> 0	If < 0: Negative samples too similar to positive - adjust corruption rate
`expert_utilization`	0.2-0.8 per expert	If < 0.1: Expert underused - consider topology change
`betti_beta_1`	5-20	If > 30: Too many cycles - reduce edges; If < 3: Too sparse - add edges
`routing_quality`	> 0.6	If < 0.5: Trigger topology reconfiguration
`ds_api_failures`	< 5%	If > 10%: Check API keys and network connectivity

Common Patterns

Pattern 1: "Cold Start" - Bootstrap from No Data

// When starting with empty queue
PipelineConfig config;
config.enable_knowledge_engine = true;  // Enable API calls
config.acquisition_threads = 4;        // Fill queue quickly

pipeline.initialize(config);

// Wait for buffer to fill before training
while (pipeline.get_metrics().ds_queue_depth < config.batch_size * 2) {
    std::this_thread::sleep_for(std::chrono::milliseconds(100));
}

pipeline.start_training();

Pattern 2: "Adaptive Training" - Dynamic Topology

// Check metrics periodically and adjust
auto metrics = pipeline.get_metrics();

if (metrics.betti_beta_1 > 30) {
    // Too complex - simplify topology
    pipeline.trigger_topology_evaluation();
    // This will reduce edges automatically
}

Pattern 3: "Checkpoint Resume" - Fault Tolerance

// Try to resume from checkpoint
if (pipeline.import_model("checkpoint_epoch_50.bin")) {
    std::cout << "Resumed from epoch 50\n";
} else {
    std::cout << "Starting fresh\n";
}

Visual Summary: Complete Architecture

                    ┌─────────────────────────────────────┐
                    │          USER CODE / WUI            │
                    │  (Controls: start, stop, configure)  │
                    └───────────────┬─────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────────────┐
│                        AUTONOMOUS TRAINING PIPELINE                            │
├────────────────────────────────────────────────────────────────────────────────┤
│                                                                                │
│  ┌────────────────────────────────────────────────────────────────────────┐   │
  // ... (truncated)
  // See source for complete code

Quick Reference: Common Operations

Operation	Code Pattern	Expected Latency
Initialize pipeline	`pipeline.initialize(config)`	100-500ms
Get one sample	`synthesizer.get_sample()`	10-100ms (blocking)
Process batch	`pipeline.process_batch()`	50-500ms (depends on k)
Betti evaluation	`pipeline.trigger_topology_evaluation()`	10-100ms
Checkpoint	`pipeline.export_model(path)`	100-1000ms

Document Version: 1.0 | Generated for q_mini_wasm_v2 training pipeline

Guides Cognitive Ergonomics Training Pipeline - kennetholsenatm-gif/q_mini_wasm_v2 GitHub Wiki

Cognitive Ergonomics Guide: Training Pipeline Mental Model

Quick Mental Model: The "Knowledge Refinery"

Decision Flow: "Should I Trigger Betti Guidance?"

Data Flow: From API to Trained Expert

Configuration Decision Tree

Error Handling Mental Model

Metric Interpretation Guide

Common Patterns

Pattern 1: "Cold Start" - Bootstrap from No Data

Pattern 2: "Adaptive Training" - Dynamic Topology

Pattern 3: "Checkpoint Resume" - Fault Tolerance

Visual Summary: Complete Architecture

Quick Reference: Common Operations

⚠️ GitHub.com Fallback ⚠️

Guides Cognitive Ergonomics Training Pipeline - kennetholsenatm-gif/q_mini_wasm_v2 GitHub Wiki

Cognitive Ergonomics Guide: Training Pipeline Mental Model

Quick Mental Model: The "Knowledge Refinery"

Decision Flow: "Should I Trigger Betti Guidance?"

Data Flow: From API to Trained Expert

Configuration Decision Tree

Error Handling Mental Model

Metric Interpretation Guide

Common Patterns

Pattern 1: "Cold Start" - Bootstrap from No Data

Pattern 2: "Adaptive Training" - Dynamic Topology

Pattern 3: "Checkpoint Resume" - Fault Tolerance

Visual Summary: Complete Architecture

Quick Reference: Common Operations

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️