Module_DeepLearning_Audio - vorcyc/Vorcyc.Mathematics GitHub Wiki

Location: Home / Deep learning module / Audio integration

Deep learning × signal processing integration

This document summarizes the full Signal → BatchTensor → Trainer pipeline (phases A–D). Primary APIs live in Vorcyc.Mathematics.DeepLearning.Integration and Integration.Frontends.

Data layout (NHWC)

Use Shape Typical entry
Waveform batch N×1×1×L BatchTensorSignalExtensions.FromSignalVectors
Spectrogram batch N×T×F×1 SpectrogramBatchBuilder.FromSignals
Periodogram / feature vector N×1×1×F FeatureBatchBuilder.FromPeriodograms / FromExtractorMeans
Frame feature sequence N×T×F×1 FeatureBatchBuilder.FromExtractors
Classification labels N×1×1×C AudioTrainingSamples.CreateOneHotTargets
Regression scalar N×1×1×1 AudioTrainingSamples.CreateScalarTargets

Dependency direction: DeepLearning → SignalProcessing (DSP core does not reference DL).

Phase A: bridging and training samples

var sample = AudioTrainingSamples.SpectrogramClassification(
    signals, labels, numClasses: 2, stft);

trainer.FitBatchSequential(model, loss, optimizer, [sample], epochs: 100);

Phase B: feature packing

var mfcc = AudioTrainingSamples.CreateDefaultMfccExtractor(8000);
var features = FeatureBatchBuilder.FromExtractorMeans(signals, mfcc); // N×1×1×13

Phase C: network-embeddable DSP frontends

var stack = AudioFrontendLayers.CreateMelSpectrogramStack(8000, 64, 32, melBands: 24);
var model = new BatchSequential<float>(stack[0], stack[1], stack[2], /* CNN head */);
  • BatchStftMagnitudeLayer backward is stop-gradient (CNN head only).
  • BatchPreEmphasisLayer and BatchMelFilterbankLayer support gradients (linear).

Phase D: neural regression and waveform regression

For neural-network curve fitting, use MlpRegressor (DeepLearning.Training) — see Examples/DeepLearning_example/CurveFitDemo.cs. For waveform → scalar regression, build a BatchSequential and train with a batch MSE loss:

var sample = AudioTrainingSamples.WaveformRegression(signals, scalarTargets);
trainer.FitBatchSequential(model, new BatchMeanSquaredErrorLoss<float>(), optimizer, [sample], epochs: 100);

Test index

Test Description
SignalBatchTensor_test Waveform roundtrip
SpectrogramBatch_training_test Spectrogram CNN classification
FeatureBatchBuilder_test Feature shapes
MfccMean_training_test MFCC mean + MLP
AudioFrontend_training_test Waveform → frontend → CNN
WaveformScalarRegression_test Waveform → amplitude regression

See Module_DeepLearning.md for the full module manual.

⚠️ **GitHub.com Fallback** ⚠️