Location: Home / Deep learning module

Starting with 0.9, this module provides a complete differentiable training stack: tensor and layer definitions, automatic differentiation (forward/backward), loss functions, optimizers, training loops, and v3 model serialization. Aside from the early legacy static Layers utility class, the recommended APIs live under the Core / Modules / Training / Serialization namespaces.

• Single-sample tensor Tensor<T> (LinearAlgebra): width × height × channels (W×H×C), used by Sequential<T> / MLP heads.
• Batch tensor BatchTensor<T>: N×H×W×C (NHWC) contiguous memory, used for CNN batch training.
• Bridge: CnnMlpModel<T> converts backbone output N×1×1×F to head input 1×N×F.

📒Contents

• 🔖 Module architecture and training flow
• 🔖 Core: BatchTensor and model containers
• 🔖 Modules: batch layer overview
• 🔖 Training: Trainer
• 🔖 Losses and Optimizers
• 🔖 Serialization: ModelSerializer v3
• 🔖 Batch training code example
• 🔖 Signal processing integration
• 🔖 Feature batching (Phase B)
• 🔖 DSP frontend layers (Phase C)
• 🔖 Audio integration overview (Phase D)
• 🔖 Legacy: Layers static class
  • 🔖 BatchNorm method
  • 🔖 Conv2D method
  • 🔖 Dense method
  • 🔖 Fusion method
  • 🔖 JoinLayer method
  • 🔖 LinearLayer method
  • 🔖 MaxPool2D method
  • 🔖 ReLU method
  • 🔖 Sigmoid method
  • 🔖 Upsample2D method

Typical CNN classification pipeline:

1. Assemble BatchSequential<float> from BatchConvolution2DLayer, BatchBatchNormLayer, pooling/activation layers, and so on.
2. (Optional) BatchFlattenLayer + BatchFullyConnectedLayer + BatchSoftmaxLayer.
3. Build a batch list with BatchLabelSample<float>(input, target).
4. new Trainer<float>().FitBatchSequential(model, loss, optimizer, batches, epochs).
5. ModelSerializer.SaveToFile(model, path) to persist weights and architecture metadata (v3).

Hybrid model: CnnMlpModel<T> = BatchSequential backbone + Sequential head, trained with Trainer.FitCnnMlp.

Dual branch: BatchParallelConcatModel<T> runs left and right branches on the same input in parallel and concatenates along the channel dimension, trained with Trainer.FitBatchParallelConcat.

Four-dimensional NHWC batch tensor; Values is a Span<T>. Main members:

• Constructor: BatchTensor(batch, height, width, channels)
• Indexer: this[n, h, w, c]
• Shape → BatchShape; GetSample(int) exports a single-sample Tensor<T>

Differentiable layer interface: Forward(BatchTensor, training), Backward(grad), Parameters, GetOutputShape.

BatchSequential<T> stacks multiple IBatchLayer<T> instances in order and provides ZeroGradients().

Single-sample layer stack; input/output are Tensor<T> (1×N×F feature vectors or W×H×C image tensors).

• Backbone: BatchSequential<T> (NHWC)
• Head: Sequential<T> (MLP)
• End-to-end Forward / Backward / ZeroGradients

• Left / Right: two BatchSequential<T> branches
• Forward: same input, channel-dimension Concat

Trainable weight/bias wrapper with Value, Gradient, ZeroGradient().

The following types all implement IBatchLayer<T> (T is float, double, or another IBinaryFloatingPointIeee754):

Single-sample layers (ILayer<T>): FullyConnectedLayer, Convolution2DLayer, BatchNormLayer, ReLUActivation, SigmoidActivation, SoftmaxLayer, FlattenLayer, MaxPool2DLayer, and others, used by Sequential<T> and MLP heads.

Trainer<T> provides high-level training loops:

Step APIs: TrainBatchSequentialStep, TrainBatchParallelConcatStep, and others for custom loops.

Learning-rate scheduling: ILearningRateScheduler<T> — optional callback at epoch start (also used by MlpTrainingOptions<T>).

BatchTrainingSample<T> — record struct for FitCnnMlp: NHWC Input + legacy 1×N×F Target tensor.

MlpTrainingOptions<T> — curve-fitting MLP training: OptimizerKind (Sgd/Adam), LR, momentum, Adam betas, optional LearningRateScheduler, RandomSeed, ComputingContext; CreateOptimizer() builds IOptimizer<T>.

MlpRegressor — builds/trains small MLPs for neural-network regression (DeepLearning.Training); see Examples/DeepLearning_example/CurveFitDemo.cs.

Fit, FitBatched, FitBatchSequential, FitBatchParallelConcat, and FitCnnMlp all accept an optional computingContext parameter (or use an outer ComputingScope). Batch layer forwards read the execution policy; see the computing context guide for details.

IBatchLoss<T>: Compute(prediction, target) + Backward(...), gradient shape matches BatchTensor.

• SgdOptimizer<T>: stochastic gradient descent (optional momentum)
• AdamOptimizer<T>: Adam

Both implement IOptimizer<T>: ZeroGrad, Step(parameters).

File magic VMDL, FormatVersion = 3. In addition to tensor weights, writes layer type IDs and hyperparameters (LayerDescriptor); on load, ModelArchitectureCatalog.VerifyMatches can validate structure.

Supported model kinds (ModelKind):

API example:

ModelSerializer.SaveToFile(model, "model.vmdl");
ModelSerializer.LoadFromFile(model, "model.vmdl"); // Model structure must be constructed with the same architecture first

ModelArchitectureCatalog.Describe(model) exports a layer descriptor list for debugging or unit tests.

Composite layers such as BatchResidualBlockLayer and BatchSqueezeExciteLayer wrap child layers (e.g. BatchBatchNormLayer) internally. Before 0.9.2, serialization only walked the top-level Parameters and missed the running mean/variance of nested BatchNorm (inference statistics are not trainable Parameters, yet must be persisted — otherwise inference drifts after reload).

The fix introduces the IBatchCompositeLayer<T> interface:

public interface IBatchCompositeLayer<T>
    where T : unmanaged, IBinaryFloatingPointIeee754<T>
{
    // Ordered child layers nested inside this composite, so the serializer can recurse
    // into their non-parameter state (e.g. BN running statistics).
    IReadOnlyList<IBatchLayer<T>> Children { get; }
}

When writing/reading format v3, ModelSerializer recurses into child layers of any layer implementing this interface, persisting each child's tensor state (including BN running stats). Custom composite layers holding stateful children should implement it, otherwise their child state will not be persisted.

ModelArchitectureBuilder — rebuild trainable models from LayerDescriptor lists:

LayerBase<T> / BatchLayerBase<T> — optional base classes for custom layers.

BatchTensorUtilities / TensorUtilities — shape checks, conversions, debug helpers.

using Vorcyc.Mathematics.DeepLearning;
using Vorcyc.Mathematics.DeepLearning.Losses;
using Vorcyc.Mathematics.DeepLearning.Modules;
using Vorcyc.Mathematics.DeepLearning.Optimizers;
using Vorcyc.Mathematics.DeepLearning.Training;
using Vorcyc.Mathematics.DeepLearning.Serialization;

// Simple CNN: Conv → BN → ReLU → GAP → FC → Softmax
var model = new BatchSequential<float>(
    new BatchConvolution2DLayer<float>(3, 16, kernelSize: 3, stride: 1, name: "conv1"),
    new BatchBatchNormLayer<float>(16, name: "bn1"),
    new BatchReLUActivation<float>("relu1"),
    new BatchGlobalAveragePool2DLayer<float>("gap"),
    new BatchFullyConnectedLayer<float>(16, 10, name: "fc"),
    new BatchSoftmaxLayer<float>("softmax"));

var batches = new List<BatchLabelSample<float>>(); // Fill with NHWC input / target
var trainer = new Trainer<float>();
var loss = new BatchCategoricalCrossEntropyLoss<float>();
var optimizer = new AdamOptimizer<float>(learningRate: 1e-3f);

trainer.FitBatchSequential(model, loss, optimizer, batches, epochs: 10,
    onEpochEnd: (epoch, avgLoss) => Console.WriteLine($"epoch {epoch}: {avgLoss}"));

ModelSerializer.SaveToFile(model, "cnn.vmdl");

DeepLearning.Integration converts DSP Signal values to BatchTensor<float> for direct use with Trainer.FitBatchSequential. Dependency direction is DeepLearning → SignalProcessing (the signal module does not reference deep learning).

using Vorcyc.Mathematics.DeepLearning.Integration;
using Vorcyc.Mathematics.DeepLearning.Training;
using Vorcyc.Mathematics.SignalProcessing.Signals;
using Vorcyc.Mathematics.SignalProcessing.Fourier;

var low = new Signal(256, 8000f);
low.GenerateWave(WaveShape.Sine, 200f);
var highTone = new Signal(256, 8000f);
highTone.GenerateWave(WaveShape.Sine, 600f);

var stft = new Stft(windowSize: 64, hopSize: 32);
var spec = SpectrogramBatchBuilder.FromSignal(low, stft);            // 1×T×F×1
var vectorBatch = BatchTensorSignalExtensions.FromSignal(low);       // 1×1×1×L
var restored = vectorBatch.ToSignal(0, 8000f);                     // Round-trip waveform

var sample = AudioTrainingSamples.SpectrogramClassification(
    signals: [low, highTone],
    classIndices: [0, 1],
    numClasses: 2,
    stft);

trainer.FitBatchSequential(model, loss, optimizer, [sample], epochs: 100);

FeatureBatchBuilder standardizes DSP feature extraction results as DL tensors:

AudioTrainingSamples provides training sample factories such as PeriodogramClassification, FeatureMeanClassification, and MfccSequenceClassification.

var mfcc = AudioTrainingSamples.CreateDefaultMfccExtractor(8000);
var sample = AudioTrainingSamples.FeatureMeanClassification(
    signals, labels, numClasses: 2, extractor: mfcc);

Integration.Frontends wraps fixed DSP operations as IBatchLayer<float> layers that can be composed into BatchSequential:

using Vorcyc.Mathematics.DeepLearning.Integration.Frontends;

var frontend = AudioFrontendLayers.CreateMelSpectrogramStack(
    samplingRate: 8000, windowSize: 64, hopSize: 32, melBands: 24);

var model = new BatchSequential<float>(
    frontend[0], frontend[1], frontend[2],
    new BatchConvolution2DLayer<float>(1, 16, kernelSize: 3),
    new BatchReLUActivation<float>(),
    new BatchGlobalAveragePool2DLayer<float>(),
    new BatchFullyConnectedLayer<float>(16, numClasses));

var waves = BatchTensorSignalExtensions.FromSignalVectors(signals);
trainer.FitBatchSequential(model, loss, optimizer,
    [new BatchLabelSample<float>(waves, targets)], epochs: 100);

• Neural regression: for neural-network curve fitting use MlpRegressor (DeepLearning.Training) — see Examples/DeepLearning_example/CurveFitDemo.cs. (CurveFitter<T> itself lives in MachineLearning.CurveFitting and covers classical curve fitting, not neural networks.)
• Batch regression loss: BatchMeanSquaredErrorLoss<T>, paired with AudioTrainingSamples.WaveformRegression.
• Documentation and examples: see Module_DeepLearning_Audio.md.

The following are early non-autodiff parallel forward operators. New projects should prefer Modules + Trainer. Retained for compatibility and reference.

Vorcyc.Mathematics.DeepLearning.Layers.Layers is a static class that implements neural network layers.

• public static Tensor<T> BatchNorm<T>(Tensor<T> input, Tensor<T> mean, Tensor<T> variance, Tensor<T> shift, Tensor<T> scale) where T : IBinaryFloatingPointIeee754<T>

  • Applies batch normalization to the input tensor.
  • Parameters:
    • input: Input tensor.
    • mean: Mean tensor.
    • variance: Variance tensor.
    • shift: Shift tensor.
    • scale: Scale tensor.
  • Returns: Normalized tensor.

• public static TensorFp32 BatchNorm(TensorFp32 input, TensorFp32 mean, TensorFp32 variance, TensorFp32 shift, TensorFp32 scale)

  • Applies batch normalization to the input tensor.
  • Parameters:
    • input: Input tensor.
    • mean: Mean tensor.
    • variance: Variance tensor.
    • shift: Shift tensor.
    • scale: Scale tensor.
  • Returns: Normalized tensor.

• public static Tensor<T> Conv2D<T>(Tensor<T> input, Tensor<T>[] filters, Tensor<T> biases, int stride = 1, int dilation = 1) where T : IBinaryFloatingPointIeee754<T>

  • Performs convolution on a 2D tensor.
  • Parameters:
    • input: Input 2D tensor.
    • filters: Array of convolution kernels.
    • biases: Bias tensor.
    • stride: Convolution stride; default is 1.
    • dilation: Convolution dilation rate; default is 1.
  • Returns: Convolution result tensor.

• public static TensorFp32 Conv2D(TensorFp32 input, TensorFp32[] filters, TensorFp32 biases, int stride = 1, int dilation = 1)

  • Performs convolution on a 2D tensor.
  • Parameters:
    • input: Input 2D tensor.
    • filters: Array of convolution kernels.
    • biases: Bias tensor.
    • stride: Convolution stride; default is 1.
    • dilation: Convolution dilation rate; default is 1.
  • Returns: Convolution result tensor.

• public static Tensor<T> Dense<T>(Tensor<T> input, Tensor<T>[] weights, Tensor<T> biases) where T : IBinaryFloatingPointIeee754<T>

  • Applies a dense (fully connected) layer to the input tensor.
  • Parameters:
    • input: Input tensor.
    • weights: Array of weight tensors.
    • biases: Bias tensor.
  • Returns: TensorFp32 after the dense operation.

• public static TensorFp32 Dense(TensorFp32 input, TensorFp32[] weights, TensorFp32 biases)

  • Applies a dense (fully connected) layer to the input tensor.
  • Parameters:
    • input: Input tensor.
    • weights: Array of weight tensors.
    • biases: Bias tensor.
  • Returns: TensorFp32 after the dense operation.

• public static Tensor<T> Fusion<T>(Tensor<T> input, Tensor<T> joint) where T : IBinaryFloatingPointIeee754<T>

  • Merges two tensors.
  • Parameters:
    • input: Input tensor.
    • joint: TensorFp32 to merge.
  • Returns: Merged tensor.

• public static TensorFp32 Fusion(TensorFp32 input, TensorFp32 joint)

  • Merges two tensors.
  • Parameters:
    • input: Input tensor.
    • joint: TensorFp32 to merge.
  • Returns: Merged tensor.

• public static Tensor<T> JoinLayer<T>(Tensor<T> input, Tensor<T> joint) where T : IBinaryFloatingPointIeee754<T>

  • Joins two tensor layers.
  • Parameters:
    • input: Input tensor.
    • joint: TensorFp32 to join.
  • Returns: Joined tensor.

• public static TensorFp32 JoinLayer(TensorFp32 input, TensorFp32 joint)

  • Joins two tensor layers.
  • Parameters:
    • input: Input tensor.
    • joint: TensorFp32 to join.
  • Returns: Joined tensor.

• public static Tensor<T> LinearLayer<T>(Tensor<T> input, Tensor<T>[] weights, Tensor<T> biases) where T : IBinaryFloatingPointIeee754<T>

  • Applies a linear (fully connected) layer to the input tensor.
  • Parameters:
    • input: Input tensor.
    • weights: Array of weight tensors.
    • biases: Bias tensor.
  • Returns: TensorFp32 after the linear operation.

• public static TensorFp32 LinearLayer(TensorFp32 input, TensorFp32[] weights, TensorFp32 biases)

  • Applies a linear (fully connected) layer to the input tensor.
  • Parameters:
    • input: Input tensor.
    • weights: Array of weight tensors.
    • biases: Bias tensor.
  • Returns: TensorFp32 after the linear operation.

• public static Tensor<T> MaxPool2D<T>(Tensor<T> input) where T : IBinaryFloatingPointIeee754<T>, IMinMaxValue<T>

  • Applies 2D max pooling to the input tensor.
  • Parameters:
    • input: Input tensor.
  • Returns: TensorFp32 after max pooling.

• public static TensorFp32 MaxPool2D(TensorFp32 input)

  • Applies 2D max pooling to the input tensor.
  • Parameters:
    • input: Input tensor.
  • Returns: TensorFp32 after max pooling.

• public static Tensor<T> ReLU<T>(Tensor<T> input) where T : IBinaryFloatingPointIeee754<T>

  • Applies the ReLU activation function to the input tensor.
  • Parameters:
    • input: Input tensor.
  • Returns: TensorFp32 after ReLU.

• public static TensorFp32 ReLU(TensorFp32 input)

  • Applies the ReLU activation function to the input tensor.
  • Parameters:
    • input: Input tensor.
  • Returns: TensorFp32 after ReLU.

• public static Tensor<T> Sigmoid<T>(Tensor<T> input) where T : IBinaryFloatingPointIeee754<T>

  • Applies the Sigmoid activation function to the input tensor.
  • Parameters:
    • input: Input tensor.
  • Returns: TensorFp32 after Sigmoid.

• public static TensorFp32 Sigmoid(TensorFp32 input)

  • Applies the Sigmoid activation function to the input tensor.
  • Parameters:
    • input: Input tensor.
  • Returns: TensorFp32 after Sigmoid.

• public static Tensor<T> Upsample2D<T>(Tensor<T> input) where T : IBinaryFloatingPointIeee754<T>

  • Applies 2D upsampling to the input tensor.
  • Parameters:
    • input: Input tensor.
  • Returns: Upsampled tensor.

• public static TensorFp32 Upsample2D(TensorFp32 input)

  • Applies 2D upsampling to the input tensor.
  • Parameters:
    • input: Input tensor.
  • Returns: Upsampled tensor.

The following example uses several methods from the Layers class, with comments in the sample:

using System;
using Vorcyc.Mathematics.DeepLearning.Layers;
using Vorcyc.Mathematics.LinearAlgebra;

public class LayersExample
{
    public static void Main()
    {
        // Define input tensor
        var input = new Tensor<float>(new float[,,] 
        { 
            { 
                { 1.0f, 2.0f }, 
                { 3.0f, 4.0f } 
            }, 
            {
                { 5.0f, 6.0f }, 
                { 7.0f, 8.0f }
            } 
        });
        // Define mean, variance, shift, and scale tensors
        var mean = new Tensor<float>(new float[] { 1.0f, 2.0f });
        var variance = new Tensor<float>(new float[] { 1.0f, 1.0f });
        var shift = new Tensor<float>(new float[] { 0.0f, 0.0f });
        var scale = new Tensor<float>(new float[] { 1.0f, 1.0f });

        // Apply batch normalization
        var normalized = Layers.BatchNorm(input, mean, variance, shift, scale);
        Console.WriteLine("BatchNorm Result:");
        PrintTensor(normalized);

        // Define convolution kernels and bias tensor
        var filters = new Tensor<float>[]
        {
        new Tensor<float>(new float[,,] { { { 1.0f, 0.0f }, { 0.0f, 1.0f } } }),
        new Tensor<float>(new float[,,] { { { 0.0f, 1.0f }, { 1.0f, 0.0f } } })
        };
        var biases = new Tensor<float>(new float[] { 0.0f, 0.0f });

        // Apply convolution
        var convResult = Layers.Conv2D(input, filters, biases);
        Console.WriteLine("Conv2D Result:");
        PrintTensor(convResult);

        // Define weight and bias tensors
        var weights = new Tensor<float>[]
        {
        new Tensor<float>(new float[,,] { { { 1.0f, 0.0f }, { 0.0f, 1.0f } } }),
        new Tensor<float>(new float[,,] { { { 0.0f, 1.0f }, { 1.0f, 0.0f } } })
        };
        var denseBiases = new Tensor<float>(new float[] { 0.0f, 0.0f });

        // Apply dense layer
        var denseResult = Layers.Dense(input, weights, denseBiases);
        Console.WriteLine("Dense Result:");
        PrintTensor(denseResult);

        // Apply ReLU activation
        var reluResult = Layers.ReLU(input);
        Console.WriteLine("ReLU Result:");
        PrintTensor(reluResult);

        // Apply Sigmoid activation
        var sigmoidResult = Layers.Sigmoid(input);
        Console.WriteLine("Sigmoid Result:");
        PrintTensor(sigmoidResult);
    }

    private static void PrintTensor(Tensor<float> tensor)
    {
        for (int d = 0; d < tensor.Depth; d++)
        {
            for (int y = 0; y < tensor.Height; y++)
            {
                for (int x = 0; x < tensor.Width; x++)
                {
                    Console.Write($"{tensor[x, y, d]} ");
                }
                Console.WriteLine();
            }
            Console.WriteLine();
        }
    }
}