ML Pipelines

Overview

ix-pipeline provides DAG-based workflow composition for ML tasks. Pipelines are the primary way to orchestrate ix crates into complete ML workflows.

Pipeline Architecture

Input Data → Preprocessing → Feature Engineering → Model Training → Evaluation → Output
     ↓              ↓                ↓                   ↓              ↓
  ix-io         ix-math          ix-math           ix-supervised    ix-math
                ix-signal                          ix-unsupervised
                                                   ix-nn
                                                   ix-ensemble

Pipeline Types

Ephemeral Pipelines

• Created on-the-fly for one-time tasks
• No persistent state
• Auto-cleaned after completion

Persistent Pipelines

• Saved configurations for recurring tasks
• Cached intermediate results via ix-cache
• Versioned and tracked

Governance Feedback Pipelines

Special ix ML pipelines that analyze Demerzel's governance data and return calibration recommendations. This creates a compounding loop: Demerzel governs ix, ix improves Demerzel's governance.

Confidence Calibrator

• Input: Historical belief states with truth value changes
• Output: Calibration report (overconfidence rate, threshold recommendations)
• Frequency: Weekly
• Guardrail: Threshold adjustments cannot exceed +/- 0.1 per cycle

Staleness Predictor

• Input: Belief timestamps, evidence freshness, topic volatility
• Output: At-risk beliefs, recommended recon targets
• Frequency: Daily
• Guardrail: Cannot trigger more than 3 proactive recons per day

Violation Pattern Detector

• Input: Historical compliance reports across all repos
• Output: Systemic patterns, one-off incidents, emerging risks
• Frequency: Weekly
• Guardrail: Cannot auto-create policies -- only recommends

Remediation Optimizer

• Input: PDCA records (actions, outcomes, completion rates)
• Output: Success rates by strategy, automation candidates
• Frequency: Monthly
• Guardrail: Cannot downgrade high-risk gaps without human approval

Anomaly Detector

• Input: All governance exports (beliefs, evolution, compliance)
• Output: Anomaly alerts with severity
• Frequency: Real-time
• Guardrail: Cannot trigger autonomous action for anomalies with confidence < 0.7

Bootstrapping the Feedback Loop

The ML feedback loop has four phases:

Phase	Duration	Description
1. Data Collection	Days 1-30	Demerzel exports data, no ML recommendations applied
2. Shadow Mode	Days 31-60	ML produces recommendations but Demerzel only logs them
3. Supervised	Days 61-90	Low-risk recommendations applied with human notification
4. Autonomous	Day 91+	Full feedback loop within guardrails, quarterly human review

Using the ix-ml-builder Skill

The /ix-ml-builder Claude Code skill provides an interactive way to build ML pipelines:

/ix-ml-builder classification --data my_data.csv --target label

The skill auto-detects task type, selects appropriate models, handles preprocessing, evaluation, and caching.

Crate Dependencies for ML

ix-pipeline
├── ix-supervised (regression, classification, SVM, trees)
├── ix-unsupervised (k-means, DBSCAN, PCA, t-SNE)
├── ix-nn (feedforward, CNN, RNN, transformers)
├── ix-ensemble (random forest, gradient boosting, stacking)
├── ix-probabilistic (Bayesian, HMM, Gaussian processes)
├── ix-math (linear algebra, statistics, calculus)
├── ix-cache (result caching, memoization)
└── ix-io (data loading, format conversion)