This page is the learning half: the layers, the losses, the optimizers and the loop that trains them. A network can be described in Keras's words, written as code, or declared in the chain of the pipeline that prepares its rows — it is the same network whichever way it is written. Trained behind a pipeline, it is measured by the pipeline's report, drawn, and kept with that pipeline as one file.

The rule to remember: the network learns from the training rows alone, is judged on the validation rows once an epoch, and never sees the test rows until the report measures them.

The rest of this page explains each one, starting with a whole run and then each part in turn.

The layers and the loop live in DeepSharp, beside the tensors. Where a network meets a pipeline is DeepSharp.Learners.Networks, and the charts are DeepSharp.Charts. Each is a package of its own, so a project carries only what it uses.

dotnet add package DeepSharp
dotnet add package DeepSharp.Pipelines
dotnet add package DeepSharp.Learners.Networks
dotnet add package DeepSharp.Charts

The arithmetic runs on the light engine unless you hand it another, such as libtorch — TorchSharp backend. A network PyTorch, Keras or an ONNX exporter saved is read into the same network — Importing a model.

using DeepSharp.Charts;
using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    // ---- nothing above this line learns from the rows ----
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")      // a network takes every feature between -1 and 1: the default
    .Report(report => report
        .Measure(Metric.Accuracy, Metric.Precision, Metric.Recall, Metric.ConfusionMatrix)
        .On(Part.Train, Part.Validation, Part.Test)
        .As(Shown.Numbers, Shown.Drawn))
    .WithTensorflow(network => network                // and the network those rows train
        .Dense(16).Relu().Dense(1)
        .Adam(0.01)
        .BinaryCrossEntropy()
        .Run(seed: 20260929, epochs: 100)
        .StoppingAfter(patience: 10))
    .Build();

var trained = pipeline.Train();

var test = trained.Measures!.Parts[^1];               // the test rows' measures, each beside the average's
File.WriteAllText("titanic.network.json", trained.ToJson());
File.WriteAllText("titanic-loss.svg", trained.History!.LossCurve());

The network learns from the training rows and is judged by the validation rows once an epoch. The test rows never reach the loop; the report measures them once it has learned.

Run on the published file, it stopped after 46 epochs, kept the network of its 36th, and got 0.815 of the test passengers right, where predicting the training rows' most common answer gets 0.615.

The networks sample in the repository trains it, a price five days on and a day's bikes hour by hour, and prints what each did.

• In Keras's words — a Sequential and its layers.
• As code — a LayerStack, or a class of your own derived from Network.
• In the chain of the pipeline that prepares the rows — it has a section of its own, below.

All three lower to the same network, number for number.

new Sequential() and then the layers:

• a dense layer: .Dense(units);
• the activations: .Relu(), .Tanh(), .Sigmoid(), .Gelu(), .Silu(), .Mish(), .LeakyRelu(), .PRelu(), .Elu(), .Selu(), .Softmax();
• dropout and the normalisations: .Dropout(rate), .BatchNorm(), .LayerNorm(), .GroupNorm(groups), .InstanceNorm(), .RmsNorm();
• a number or a category as more than one value: .Periodic(periodicity), .Embedding(block, width);
• the shape: .Reshape(shape), .Flatten();
• for a series, an image or a volume: .Conv1D(filters, window), .Conv2D(filters, window, groups), .Conv3D(filters, window), .DepthwiseConv2D(multiplier, window), .Conv2DTranspose(filters, window), .Pad2D(border), .MaxPool2D(2), .AvgPool1D(window), .AdaptiveAvgPool2D(rows, columns), .GlobalAvgPool2D() and .SpatialDropout2D(rate) — set out below, under A series, an image, a volume.

Nothing is said about widths. Each is worked out from the shape of an example as it reaches the layer: a row of numbers for a dense layer, an image of rows, columns and channels for a convolution.

A layer that starts otherwise or has no bias takes one more argument, .Dense(units, new Beginning { WithoutBias = true }). One that carries a penalty, a bound or a hold is followed by the word that says it, .Dense(units).Ruled(new SlotRules { L2 = 0.0001 }) or .Dense(units).Held(). Both are set out below, under How a layer starts and Penalising, bounding and holding still.

A word an example cannot reach is refused, naming the word, where it stands and what reached it:

This network cannot be lowered at 36: word 2, dense, takes each example as a row of numbers, and each reaching it is 6x6x1: flatten it first.

Compile takes the description as it stands and draws nothing. The network is built at its first fit, from the shape of the rows it learns from and the run's seed. So the one seed the history records reproduces the whole run: what the layers start at, the order of every epoch's rows, what every dropout leaves out.

• .Input(shape) states the shape of an example, so every word is checked where the description is compiled.
• To hold the network before any rows, lower it by hand — description.Lower(new Shape(14), new RandomStream(42)) — and compile that.

The words take Keras's settings where Keras's differ:

• .BatchNorm(momentum, epsilon) takes Keras's momentum — the share of the running statistics a training batch leaves as they were, 0.99 in Keras — and keeps its complement, with Keras's epsilon.
• .LayerNorm(epsilon) takes Keras's epsilon.
• Without settings, both are PyTorch's.
• A window padded as TensorFlow's padding='same' is new Window(3, 3) { Stride = 2, PaddingMode = PaddingMode.Same }.

A last .Sigmoid() before BinaryCrossEntropy, or a last .Softmax() before CrossEntropy — Keras's habit — is lifted into the loss, which applies that activation itself. The network ends at the layer before it, so every prediction goes through it once, not twice. A stack that ends in such a layer is refused where it is compiled with such a loss, naming the layer.

A network is a layer that holds layers, and a LayerStack holds them in order:

using DeepSharp.Networks;

var stream = new RandomStream(42);
var network = new LayerStack(
    new Dense(14, 16, stream.Draw("initialise:0", 0, 0)),
    new Relu(),
    new Dense(16, 1, stream.Draw("initialise:2", 0, 0)));

That is exactly what the words above lower to under the same stream: the same numbers, bit for bit, and the same file.

A network with a forward pass of its own derives from Network, adds its layers by name with AddLayer and says what it does in Compute. To be saved in its file, it says the name the file knows it by and how it is built again from what it wrote:

using System.Text.Json;
using DeepSharp.Networks;
using DeepSharp.Tensors;

public sealed class Passenger : Network, ISaved<Passenger>
{
    private readonly Dense _hidden;
    private readonly Dense _out;

    public Passenger(Draws draws)
    {
        _hidden = AddLayer("hidden", new Dense(14, 16, draws));
        _out = AddLayer("out", new Dense(16, 1, draws));
    }

    public static string Name => "passenger";                   // what its file calls it

    public static Passenger Rebuild(JsonElement settings, Rebuilding rebuilding) => new(rebuilding.Draws);

    public void WriteSettings(Utf8JsonWriter writer)             // nothing: its layers write their own
    {
    }

    protected override Tensor Compute(Tensor input, Pass pass) =>
        _out.Forward(pass.Backend.Relu(_hidden.Forward(input, pass)), pass);
}

A program that reads its file back registers it with the catalog it reads with: TrainedNetwork.FromJson(text, NetworkCatalog.BuiltIn().Register<Passenger>(), StepCatalog.BuiltIn()). One that is not registered is refused, naming it, rather than read as something else.

Every number a network holds is a slot with a dotted path — hidden.weight, 0.bias, 2.running_mean — its own first, then those of the layers it holds.

• Slots() lists them, and HeldLayers() every layer a layer holds, each with its path, so the layer holding any slot is named by the slot's path.
• A Parameter is moved by the optimizer.
• A RunningStatistic, such as a batch normalisation's mean, is moved only by a training pass, so the validation rows never shape what the network keeps.

A pass says whether it trains or evaluates, which is why no layer has a mode to switch.

This is the door the page opened with, written out. The chain that prepares the rows names the network they train, below its output:

• .WithTensorflow(…) in the words TensorFlow and Keras use;
• .WithTorch(…) in the words PyTorch uses.

pipeline.Train() runs the pipeline and trains exactly that network. A network written beside its pipeline and fitted by hand is the other door, and the sections below it are about both:

using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("btceur-1d.csv")                                // declared, not opened
    .Declare(schema => schema
        .Timestamp("timestamp")
        .Number("close")
        .Optional("trades", ColumnKind.Number))              // a column that may have gaps
    .OrderBy("timestamp")
    .SettleGaps(gaps => gaps.Zero("trades"))                 // the gaps first: a quiet day traded nothing
    .AddFeature("turnover", "close", Arithmetic.Times, "trades") // then the features, worked out from settled columns
    .SplitByTime("timestamp", train: 0.70, validation: 0.15, gap: 1) // then the split: test is the rest
    .Ahead("close", 1, AheadAs.Return)                       // the answer: tomorrow's return
    .Drop("timestamp")                                       // what no model reads, before what learns
    .Normalise(scale => scale                                // and below it what learns from the training rows alone
        .Columns("close", "turnover")                        // where this pipeline's features land: -1 to 1 by default
        .MaxAbs("trades"))                                   // a count, divided by its largest, so a quiet day stays nought
    .Report(report => report.Measure(Metric.Rmse).On(Part.Validation, Part.Test).As(Shown.Numbers))
    .WithTensorflow(network => network                       // and the network these rows train
        .Dense(8).Relu().Dense(1)                            // Keras's words: the widths come from the rows
        .Adam()
        .MeanSquaredError()
        .Run(seed: 42, epochs: 20))
    .Build();

var trained = pipeline.Train();                              // the course, from the file to the trained model

var file = trained.ToJson();                                 // the network and its pipeline, one file

btceur-1d.csv is a file of your own, and none ships here: a row a day of bitcoin's price in euros, under a header that names timestamp, close and trades. Live sources lands a file with those columns among its ten.

The network is one more step of the declaration, so the file says what was trained behind its steps, and training it again from the file gives the same network, number for number.

Train():

1. runs the pipeline for the learner the declaration names;
2. trains the network;
3. measures it as the pipeline's report declares;
4. hands back the same TrainedNetwork a network written as code gives — a test holds the declared network and the hand-written one to the same numbers.

prepared.Train() does it for a run already in hand. A pipeline that names no learner says so rather than guessing one. Pipeline has the step as a file holds it, and a block of the Notebook holds the same step.

They are the ones a network's own file writes, so a declaration and the network it trains say dense and adam alike and nothing is translated between them. Each takes the settings it takes in that file:

The ones a chain names and the ones it leaves unsaid alike are written. .Run(seed) writes a hundred epochs and batches of thirty-two, and a chain that never says when to stop goes on to its last pass. A file that leaves a setting out is refused where it is read.

The exception is every setting added in 0.10.0, which the table above marks as left out when it says nothing new:

• a dense layer with a bias;
• a normalisation with a scale and a shift;
• an optimizer with no decay;
• a run that clips no value, steps after every batch and keeps no average.

They mean what the layer or the run always was, so a file that leaves them out is read as that, and a chain that says them at that meaning writes the file it always wrote. Said at any other meaning they are written, and the file is then one 0.9.0 refuses by its number.

The same words say it all at once, in the Titanic pipeline's chain:

• a first layer with no bias and a small penalty on its weights;
• a normalisation after it;
• momentum with Nesterov's look-ahead and a weight decay;
• a run that takes each step from four batches, clips each gradient value and keeps an average of the weights.

using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")
    .WithTorch(network => network                     // PyTorch's words: the options below are PyTorch's
        .Dense(16, bias: false).L2(0.0001)            // no bias of its own, and a penalty on its weights
        .BatchNorm()                                  // its shift makes up for the bias that is not there
        .Relu()
        .Dense(1)
        .Sgd(0.05, momentum: 0.9).Nesterov().WeightDecay(0.0001)
        .BinaryCrossEntropy()
        .ClipValue(1).StepEvery(4).Average(0.99)      // the run: each value clipped, a step from four batches, an average
        .Run(seed: 20260929, epochs: 100))
    .Build();

var trained = pipeline.Train();                       // the network the average of its weights ends as

The words TensorFlow and Keras use and the words PyTorch uses name the same layers, optimizers and losses. Both run on the light engine and on libtorch alike: a vocabulary is a way of speaking, never an engine.

They differ in the number each writes where a chain says none, because the two libraries leave different ones:

So the two write different files, which is right, since they are different runs. The file names no vocabulary: it says which run was asked for rather than which library asked for it, and nothing would read a name.

A pipeline hands over a flat row of numbers, so the layers that work on a row are here and the ones that work on an image — a window over it, a reshaping of it — are not. A network that takes images is written as code and trained on data a pipeline did not prepare.

The step names the engine its run happens on — light unless said, .On("torch") in the chain. The application says what the name stands for before the run:

• new Engines().Use("torch", TorchBackend.OnCpu()) gives the name an engine;
• pipeline.Train(engines) trains on it.

So one declared network runs on the light engine on a machine with nothing installed and on libtorch where libtorch is there, without a word of the file changing — TorchSharp backend has the engine. Which device it works on is the application's to say.

A name nothing was given is refused by name, with what to do, rather than answered by an engine that would quietly give other numbers.

trained.ToJson() writes the network and its pipeline as it always did, and the pipeline carries the step. So a program that reads the file teaches its catalog the verb — TrainedNetwork.FromJson(text, NetworkCatalog.BuiltIn(), StepCatalog.BuiltIn().WithNetworks()) — and a catalog that was never taught says which package brings it.

Each starts as PyTorch's does: weights uniform by the number of values that reach them, biases within one over its square root. Each forward pass and its gradients were matched against PyTorch's on the same rows.

That is what a layer starts as when nothing is said, and it is what it was in every earlier release. The next sections say how to start it another way, or without a bias.

• Relu is where to start, and still the right word for most rows of a table.
• Gelu and Silu are smooth where the rectifier bends, which deep stacks and anything shaped like a transformer were tuned on. Reach for one when you are reproducing such a model, or when a deep stack of Relu leaves units at nought for good.
• LeakyRelu and PRelu keep a sliver of a negative value, for the same trouble at less cost; PRelu learns that sliver, one share or one for each channel.
• Selu keeps a stack of dense layers normalised by itself, if its numbers start as its authors start them and no dropout but theirs stands in it.
• Softmax as a layer is for shares in the middle of a network, or at its end before a loss that does not apply one itself; before a cross-entropy it is lifted into the loss.

• BatchNorm for rows of a table and batches that are not tiny.
• LayerNorm when a batch is small or a row should be judged on its own.
• GroupNorm for images in small batches, where a batch's statistics are noise.
• InstanceNorm when each image's own contrast is what to take away.
• RmsNorm is the cheaper layer normalisation large models use: it scales and never centres.

• Periodic followed by Relu is Gorishniy et al.'s PLR, which lets a network over a table see a number's shape the way a tree sees its splits. Its sigma is the setting it turns on, and the authors searched it for every dataset, so search it — Searching for a network.
• encode.piecewise in the pipeline is the same idea without anything learned, the bins placed by the training rows' quantiles.
• Embedding gives a category a small vector instead of a column of its own for every category, for a category of many values where a column each is wide and sparse.

A layer's start can be told three things: how its weights begin, how its bias begins, and that it has no bias. They travel together as a Beginning:

• in a layer's constructor it takes the place the draws had — new Beginning(draws) is the start a layer always had;
• in a description, where the run supplies the draws, it is new Beginning { … } with none.

The weights are drawn first and the bias second whatever starts them, so a layer that changes how its bias starts starts its weights where it did. A start never reaches a file: a file holds the numbers a layer learned, not how it began.

A word of a declaration can name the starts that need no number of their own — weights: InitialiserWord.GlorotUniform, biasStart: InitialiserWord.Zeros, and kaimingUniform, fanInUniform, ones, normal, truncatedNormal and orthogonal. A constant or a spread of your own is code.

A dense layer in Keras's words starts as PyTorch's does, as it always did, because a different default would change the numbers of every network written before. Keras's own start, Glorot's weights and a bias of nought, is asked for:

using DeepSharp.Networks;

var description = new Sequential()
    .Dense(32, new Beginning { Weights = new GlorotUniform(), Bias = new Zeros() })   // Keras's start for this layer
    .Relu()
    .Dense(16, new Beginning { WithoutBias = true })                                  // a layer with no bias at all
    .BatchNorm()                                                                       // whose shift makes up for it
    .Relu()
    .Dense(1);

var stream = new RandomStream(42);
var constant = new Dense(14, 16, new Beginning(stream.Draw("initialise:0", 0, 0)) { Bias = new Constant(0.1) });
var bare = new Dense(14, 16, new Beginning(stream.Draw("initialise:1", 0, 0)) { WithoutBias = true });

var hasBias = bare.Bias is not null;                  // false: the slot is not there, and so nothing is written for it

A layer without a bias has one set of numbers fewer, adds nothing of its own to a sum, and writes "bias": false into the file. No slot is renamed, so every other number of the network keeps its path.

Reach for it when:

• a normalisation follows the layer, whose shift makes the bias redundant;
• a model must pass through the origin;
• you are reproducing a model that was made without one (Keras's use_bias=False, PyTorch's bias=False).

It costs nothing and changes nothing else; what it does not do is make a layer cheaper to train in any way worth measuring.

The same goes for a normalisation: new Affine { WithoutScale = true } or WithoutShift keeps that one number out, as Keras's scale=False and center=False do, and leaving out both is PyTorch's affine=False.

A bias cannot be given a start it will not use: a layer that has no bias and also says how its bias begins is refused, by name.

A layer built with its numbers in hand — new Dense(weights, bias) — takes a null bias for the same thing, and a layer's Bias is null when it has none, so code that reads it asks.

Three rules can ride on a layer's numbers, each about the numbers of one slot — a dense layer's weights, a convolution's kernel, a normalisation's scale — and each said where Keras says it:

using DeepSharp.Networks;

var description = new Sequential()
    .Dense(32).Ruled(new SlotRules { L2 = 0.0001, MaxNorm = 3 })    // a penalty and a bound on this layer's weights
    .Relu()
    .Dense(16, new Beginning { WithoutBias = true })
    .Ruled(new SlotRules { L1 = 0.001 })                             // a penalty on the weights of the last layer written
    .Relu()
    .Dense(1);

// A network written as code puts the same rules on the parameters themselves, and holds a layer still with Hold().
var stream = new RandomStream(42);
var first = new Dense(14, 16, stream.Draw("initialise:0", 0, 0));
var last = new Dense(16, 1, stream.Draw("initialise:2", 0, 0));
var network = new LayerStack(first, new Relu(), last);

first.Hold();                                                        // nothing moves its numbers
last.Weight.Rules = new SlotRules { L2 = 0.0001, MaxNorm = 3 };      // a penalty and a bound on the weights of the last

They look alike and are two things.

• A penalty is Keras's regularizer. It is added to the loss, so the loss a run reports includes it — in training and in validation, so early stopping compares losses that include it. It is said for the weights of a layer or for its bias alone.
• A weight decay is PyTorch's. The optimizer adds a share of every number it moves to that number's gradient, which the loss never sees, and it reaches every number the optimizer moves, scales and biases too.

Plain Sgd under a decay of twice an L2 moves the same way. Reach for a penalty when you want the pull in the reported loss or on one slot; for a decay when you reproduce a PyTorch recipe or want it on all of them.

A held layer is excluded from the loop altogether — no gradient is worked out for it. A normalisation all of whose numbers are held reads its running statistics while training too, so a frozen normalisation behaves as it does when serving.

The held layers are part of a run: a checkpoint records them, and a run that goes on from it with another set held is refused, naming the layer. Letting a held layer learn again is a new fit, whose optimizer starts with nothing remembered for it, as in PyTorch. A file records "learns": false beside the slot.

• A bound by unit is not defined on a transposed convolution's kernel — it is laid out by the channels it takes, not by each unit it makes — and is refused there, by name.
• Nothing penalises what a layer makes (Keras's activity regularizer).
• No constraint but the largest norm is built.

A declared dense layer takes l1, l2, maxNorm and learns for its weights, and the chain has .L1(…), .L2(…), .MaxNorm(…) and .Held() for the layer written last, in both vocabularies; a declared normalisation takes learns.

The same layers walk one axis, two or three.

A window is a Window1D(length) for a series, a Window(height, width) for an image and a Window3D(depth, height, width) for a volume. Each:

• moves Stride places at a time — one number for every axis, or a number of its own for each, new Window(3, 3) { Stride = new Steps(2, 1) };
• holds cells Dilation places apart (one, neighbours);
• takes the last place whose patch overhangs the end when RoundsUp says to;
• pads in one of four ways:
  • by Padding places on every side;
  • by a Border of before and after for each axis;
  • as TensorFlow's and Keras's padding='same' does (PaddingMode.Same: as many places as the stride fits, the odd place after);
  • or — for a series — as Keras's padding='causal' (PaddingMode.Causal): the window's length less one places before the series and none after it, so no place sees what follows its own.

A series is laid out step by channel, an image row by column by channel, a volume plane by row by column by channel: channels last, as Keras keeps them.

using DeepSharp.Networks;
using DeepSharp.Tensors;

// A series of 64 steps of 3 channels, in which a bump is somewhere.
var description = new Sequential()
    .Input(new Shape(64, 3))
    .Conv1D(16, new Window1D(5) { PaddingMode = PaddingMode.Causal })   // no place sees what follows it
    .Relu()
    .MaxPool1D(2)                                                        // runs of two, not overlapping
    .SpatialDropout1D(0.2)                                               // whole channels left out while training
    .Conv1D(16, new Window1D(3) { PaddingMode = PaddingMode.Same })
    .Relu()
    .GlobalMaxPool1D()                                                   // the strongest place of each channel
    .Dense(1);

var compiled = description.Compile(new Adam(0.01), new BinaryCrossEntropy());

• A pooling written with a size, MaxPool2D(2), takes runs that do not overlap, as Keras's pooling does by default.
• Written with a window, MaxPool2D(new Window(3, 3) { Stride = 2, Padding = 1 }), it walks as that window does.
• The largest a window covers is the first of values that tie, and the gradient of a loss goes to it alone, as PyTorch's max pooling sends it.
• A window over a border sees the values of the example only: the largest is never a nought the border supplies, and an average leaves the border out of its count, as TensorFlow's and ONNX's do. PyTorch counts a border it pads with, so a network trained there is AvgPool2D(window) { CountsPadding = true }.
• A global pooling gives a row of channels, which a dense layer takes directly; KeepsAxes keeps the axes it pooled as axes of one place, as Keras's keepdims and ONNX's global pools do.

A convolution's kernel is one matrix for every axis: as many rows as the window holds values, channels in included, by as many columns as channels out, which is Keras's kernel flattened. Its start is PyTorch's.

Every value and every gradient of the convolutions and the poolings was matched against PyTorch's on every engine, to a ten-thousandth for a convolution and a hundred-thousandth for a pooling. The border PyTorch cannot say — a stride with 'same', a causal border — is worked out as TensorFlow works it out and padded by hand, so the numbers are still PyTorch's arithmetic over it.

The kinds in a network's file are conv1d, conv2d, conv3d, conv1dtranspose…3dtranspose, pad1d…3d, maxpool1d…3d, avgpool1d…3d, adaptivemaxpool1d…3d, adaptiveavgpool1d…3d, globalmaxpool1d…3d, globalavgpool1d…3d and spatialdropout1d…3d; a library older than this one refuses them by name.

A convolution of groups holds a kernel as small as its groups — the values of its window for the channels of one group, by the channels it makes — on purpose. A kernel of the full size with noughts off the diagonal would receive gradient in those noughts, and a convolution trained that way stops being grouped.

DepthwiseConv1D(multiplier, window), DepthwiseConv2D and DepthwiseConv3D are the description's words for a group of every channel making so many channels of each. Keras's separable convolution is one of those and a convolution of a window of one place after it, which is how such a file is read.

What is not built is refused where a source says it: a max pooling's storage_order, and the places a max pooling picked as a second output.

The rest of what this page used to list is built, each with its values and its gradient matched to the framework that has it, on every engine:

• a stride of its own for each axis;
• a dilated window;
• channels in groups;
• a pooling that rounds its number of places up (ceil_mode);
• a pooling to a size;
• a transposed, depthwise or separable convolution;
• a layer that only pads.

A padding can fill the places it adds from the example itself, which a border of noughts cannot: Pad1D(sides, Filling.Mirror), Pad2D(margins, Filling.Repeat), Pad3D(margins, Filling.Wrap).

using DeepSharp.Networks;
using DeepSharp.Tensors;

// A series of 64 steps of 3 channels, padded by two places at each end with the series itself, mirrored, before a window of five.
var description = new Sequential()
    .Input(new Shape(64, 3))
    .Pad1D(new Sides(2, 2), Filling.Mirror)
    .Conv1D(16, new Window1D(5))
    .Relu()
    .GlobalMaxPool1D()
    .Dense(1);

var images = new Sequential()
    .Input(new Shape(28, 28, 1))
    .Pad2D(new Margins(new Sides(1, 1), new Sides(1, 1)), Filling.Wrap)   // a picture that wraps round its edges
    .Conv2D(8, new Window(3, 3))
    .Relu()
    .Flatten()
    .Dense(10);

Every value is the example's own, copied, and the gradient of a place copied twice is the sum of what its copies did. The three were matched to PyTorch's values and gradients on every engine.

• A padding that asks for more than the example can give — a mirror of the length or more — is refused by name when the layer is shown an example.
• A value beside a filling other than the constant is refused.
• A padding with a filling is written to the file as "border"; a constant padding is written as it always was.
• ONNX's Pad is read as these: reflect, edge and wrap (from opset 19).

A loss takes the network's raw numbers and says how they are read, so there is no softmax layer to forget or to apply twice, and a prediction goes through the same activation. It is always named.

It refuses a row whose answers it could not have meant — shares that do not sum to one, a chance above one — naming the row as it was read, before anything is trained on it.

It is the loss for an answer whose columns are in an order. It adds up the predicted shares and the answer's from the first column, and the loss is the size of the difference between the two, summed over every column but the last — where both have added up to one — and averaged over the batch.

A prediction one band off costs one, and ten bands off costs ten, which a cross-entropy cannot tell apart. It is the distance Metric.Emd measures.

• A pipeline whose output does not say its columns are ordered is refused it, by name, before anything is trained.
• With a remainder on the output, the bands are compared as shares of what they hold together, and the last answer — what is left — by the size of how far its share is off. So moving a share into what is left is never taken for moving it along the order.
• The squared form of the distance was measured on synthetic flocks and ended behind the cross-entropy, so it is not built.

They are matched to PyTorch's values and gradients, and to Keras's where the loss is Keras's alone: the hinge, and the binary cross-entropy's smoothing.

A cross-entropy that softens its labels is written into the network's file as the sixth version of its part, which 0.10.0 refuses by its number; one that does not is written as it always was.

The focal loss refuses an answer between nought and one, as torchvision's does. The power it weighs a label by is worked out from the logarithm of the chance the label is given, which only for a label of nought or one is never infinite.

They are matched to PyTorch 2.14.1 step by step, on the light engine and on libtorch alike. Each remembers its slots by parameter under PyTorch's names — exp_avg, exp_avg_sq, square_avg, momentum_buffer and the steps — so a checkpoint carries them.

Where a vocabulary cannot name one honestly the door refuses: RMSprop and Nadam are declared in PyTorch's words only, because Keras's differ in arithmetic, and AdamW in Keras's words writes PyTorch's numbers.

Each does nothing unless you ask, as in PyTorch. Each is held to torch's own numbers step by step on every engine, and each is remembered in a checkpoint under PyTorch's name for it.

These are PyTorch's options, and they are declared in PyTorch's words — .WithTorch(…). A network declared in Keras's words is refused them by name, saying why: Keras's weight_decay shrinks the number itself, its SGD has no dampening, and its AMSGrad places its epsilon otherwise.

An option an optimizer does not have is refused by that optimizer. A checkpoint now also refuses a tensor its optimizer does not keep, as it already refused one it lacks, so a checkpoint of one optimizer is not gone on from under another.

Optimizer is open. A subclass says what it remembers of each parameter with MemoryOf and takes it back with Recall, which hand and take a SlotMemory. Registered with a catalog by name, it goes through a checkpoint file and on from it bit for bit. A loss of your own derives from Loss as it did.

Neither is a word of a declared network. A pipeline's file names the shipped words, because a vocabulary a file names must be one every reader has, and a network of your own is compiled and fitted by hand.

• compiled.Fit(train, validation, new FitOptions(seed)) trains on rows of tensors.
• compiled.Fit(prepared, options) trains on the rows a pipeline hands over.

The seed is asked for and never assumed. The rest is Keras's fit: one epoch and batches of thirty-two unless said, the rows shuffled afresh every epoch and the last batch kept short.

EarlyStopping { Patience, MinDelta, RestoreBest } stops the run as Keras's does, and restoring the best epoch brings back every slot. A loss that is not a finite number stops the run with the epoch and the batch it came from.

The options hold the light engine unless they name another — new FitOptions(seed) { Backend = engine } — and that engine does the whole run: the training passes, the looks at the validation rows and the report's measures.

Every batch's gradients are worked out by a recording of its pass, and only towards the slots the optimizer moves. Nobody asks how a batch's own rows moved the loss, so that way back is never worked out.

History holds what the run did: every epoch's training loss, validation loss and rate, the seed, the best epoch and why it stopped.

• GradientClip in the options — new FitOptions(seed) { GradientClip = new GradientClip(1.0) } — scales a step's gradients down to a norm together when they are over it, as PyTorch's clip_grad_norm_ does, and passes them untouched, to the last bit, when they are under. ClipValue, below, clips each value on its own.
• OnEpoch is handed each epoch as it ends, in order, on the training thread.
• Cancellation stops the run before its next batch or its next look at the validation rows by throwing an OperationCanceledException and returning nothing — no history, no best epoch put back. The network then holds what the last finished batch left it, so fitting it again goes on from there, and a checkpoint taken before is as it was.

A run that neither stops is the run without them, bit for bit.

Three more things a run can be told, all off unless asked, and all of them part of what a checkpoint records:

using DeepSharp.Networks;

var options = new FitOptions(seed: 7)
{
    Epochs = 50,
    ClipValue = 1,                           // no single gradient value beyond one, clipped first
    GradientClip = new GradientClip(5.0),    // then the norm of all of them together, if over five
    StepEvery = 4,                           // a step from the mean of four batches' gradients
    Average = 0.99,                          // and an average of the weights, which the network holds at the end
};

In a step the order is always the same:

1. the gradients of the window are averaged;
2. each value is clipped;
3. the norm is clipped;
4. the optimizer steps;
5. the average takes in the new weights.

The average is part of a checkpoint, so a run goes on from one with it to the bit. No window is open across an epoch, so a checkpoint holds none.

Checkpoints are taken every epoch, or only when the validation loss improves, and a run goes on from one exactly as if it had never stopped — on the engine it was taken on, bit for bit.

A checkpoint records what its run went under: the seed, the batch size, the early stopping, the clip, the clip of each value, the window of batches, the average, the layers that were held and the engine, with the version and the device an engine that implements INamesItsVersionAndDevice names.

Handed another engine — or another seed, batch size, early stopping, clip, window of batches, average or set of held layers — a run going on from it is refused, naming each difference, before anything is put back, since it would be another run under the same seed. Gone on from on libtorch, the networks sample's Titanic run ended with weights up to 3.0e−7 from those of the run it was taken of.

The epochs are not among them, since going on to more of them is what a resume is for:

var prepared = pipeline.Run();                       // the rows alone, for a network written here rather than declared
var compiled = new Sequential().Dense(16).Relu().Dense(1).Compile(new Adam(0.01), new BinaryCrossEntropy());

compiled.Fit(prepared, new FitOptions(seed: 7)
{
    Epochs = 50,
    Checkpoints = new Checkpoints(checkpoint => File.WriteAllText("run.json", CheckpointFile.Write(compiled, prepared, checkpoint))),
});

var resumed = CheckpointFile.Read(File.ReadAllText("run.json"), NetworkCatalog.BuiltIn(), prepared);
var more = resumed.Compiled.Fit(prepared, new FitOptions(seed: 7) { Epochs = 100, ResumeFrom = resumed.Checkpoint });

CheckpointFile.Read reads the file once, whole, before it holds it to the pipeline handed over: its network is held to the pipeline the file carries, and that to the one handed over.

A checkpoint kept under keys of your own is read with NetworkDocument.ReadCheckpoint(json, network, training, catalog), its network and its run from one reading of the text.

A checkpoint 0.4.0 wrote records neither the batch size, nor the early stopping, nor the engine, and goes on under whatever it is handed, as it did.

What both doors hand back — pipeline.Train() and Fit(prepared, options) — is a TrainedNetwork: the network, its history, how the pipeline's report measured it, and what it predicts for rows that arrive later. Predictions come in the answer's own units, through the loss's activation and the pipeline's way back:

var passengers = new InMemoryRowSource(
    ["pclass", "sex", "age", "sibsp", "parch", "fare"],
    [["3", "male", "22", "1", "0", "7.25"], ["3", "Male", "22", "1", "0", "7.25"]]);

var predictions = trained.Predict(passengers);       // the chance each passenger survived: 0.16 for the first, as it ran here
var unfamiliar = predictions.Unfamiliar;              // nothing for the first; sex_other for the second

A class is never guessed: where the line between classes lies is for the use to say.

trained.Predict(rows, engine) serves on the engine handed in, for that call alone, and trained.Predict(rows) on a light engine of its own. Either asks the engine for thirty-two rows at a time, Keras's own default, and no answer depends on which rows share its pass.

A feature that held one value on every training row taught the network nothing. The encoder keeps a place for a category the training rows never held — sex_other — and marks a cell that was empty — sex_was_missing. On the Titanic passengers those are nought on all 623 training rows, so the first layer's weights for them are still the random start's after training.

A passenger written 'Male' is answered from there, by the seed rather than by anything learned: across six seeds the same passenger was given 0.257 to 0.495 as 'Male', against 0.162 to 0.186 as 'male'.

• Unfamiliar names, for each served row, the features it moves away from the one value the training rows held.
• The report counts the rows of each part that do, as UnfamiliarRows beside Rows.
• The answer itself is left as it is: a pipeline that should answer no category its training rows never held declares EncodeCategories(unseen: Unseen.Refuse).

A network takes every feature on one scale, so it is refused a run made for a learner that does without the scales or takes categories itself — RunFor(Needs.NoScale) or RunFor(Needs.Categories), which Pipeline has — and fits behind a run of every step.

A pipeline read from its file holds no rows, and fitting behind it says so, and how to have them: run its declaration over the rows, new Pipeline(declaration, rows, folder).Run().

trained.ToJson() writes the network and the pipeline it was trained behind together, and TrainedNetwork.FromJson(text, NetworkCatalog.BuiltIn(), StepCatalog.BuiltIn()) reads them back in a program that has never seen the data — with StepCatalog.BuiltIn().WithNetworks() when the pipeline names its network.

The network's part says what it was trained on:

• its features, its answers and the output that named them;
• the seed, and the epoch its numbers come from;
• each feature that held one value on every training row;
• the SHA-256 of the pipeline's own file as it was fitted.

The same names are not the same fit: fitted again on a few more rows, a pipeline keeps every feature's name and moves the numbers it learned. So the network is read only beside the very fit it was trained behind, and refused beside any other, when it is loaded and when a run goes on. A pipeline fitted again to keep its training rows for a watch is another fit too, so a network trained before it is trained again behind it — see Measuring and drawing.

A checkpoint is the same file with what the run needs to go on, and serves as a trained network too. The file names no engine, so a network trained on one engine is read back and served on another, and every line of it ends with a line feed, on every system.

The file is one text, and .NET makes no text longer than 1,073,741,791 characters. A network's numbers take about 24 characters each, so:

The measures a network is held to are the pipeline's, declared before any number exists. .Report(...) names them, the parts they are taken on, and how they are shown:

• RMSE, MAE and R² for amounts;
• accuracy, precision, recall and the confusion matrix for classes;
• Emd, Kl and Rps for shares of a whole, Emd and Rps along an order.

Pipeline has the rules. Each measure is in the answer's own units, beside what predicting the training rows' average would score, by scikit-learn's definitions for amounts and classes and scipy's for shares. The network's measures are taken on the engine its run was handed, by the same evaluation it serves with.

trained.Measures!.Report(), in DeepSharp.Charts, is the report as it says its measures are shown — the numbers, the charts or both — as one value whose ToHtml() gives its HTML. So a notebook's C# cell that ends with it shows the report.

What the report measured can also cross as text. trained.Measures!.PredictionsToJson() writes, for each part, the key of each row and what was predicted for it, with the fit the predictions were made behind. prepared.MeasureAgain(text) measures that text again on another run of the same pipeline — refused beside any other fit, as the network's file is. That is how a notebook's report block draws a network a C# cell trained: Notebook.

DeepSharp.Charts draws them, as the text of an SVG, drawn with MatPlotLibNet:

A network's file holds the kinds it is made of — each by the name it is registered under, with the settings it is rebuilt from — every slot's numbers by its path, and its loss.

NetworkDocument writes and reads it, through a NetworkCatalog of the kinds a reader knows. A network written as code is saved by a name of its own — it implements ISaved<T> and is registered with NetworkCatalog.BuiltIn().Register<Passenger>() — and nothing is ever made from a type looked up by its name. Read back, it is the same network to the last bit.

Everything wrong with a file is said at once, each at its line and column:

(9,13): 'dens' is not a kind this catalog knows. The nearest one it knows is 'dense'.
(15,7): '0.bias' is missing: every slot of the network is written.
(17,9): '0.weight' is a 4x1 slot here, and is written as 2x4x1.

A part is written as the oldest version that says what it holds. Each of these is written only when it says something:

• a layer without a bias writes "bias": false;
• a normalisation without a scale or a shift writes "scale": false or "shift": false;
• a padding that fills from the example writes "border";
• a number with a penalty, a bound or a hold writes l1, l2, maxNorm or "learns": false beside its values;
• an optimizer with an option writes weightDecay, dampening, nesterov, amsgrad or centered;
• a run writes clipValue, stepEvery, average and the averaged numbers.

Each makes its part the fifth version, which 0.9.0 refuses by its number. A network and a run that say none of them are written as they always were, and the repository holds files the published 0.9.0 packages wrote, which every release reads, writes again and serves.

They go into a network the same way: network.Load(entries) puts each into the slot its path names, all of them or none, as PyTorch's load_state_dict does with strict=True. It refuses a missing, extra or misshapen slot in the same words as the file does.

A model PyTorch, Keras or an ONNX exporter saved is read so, into a network written here or one its file describes — Importing a model has each reader.

• A schedule that watches the validation loss. The rows a model is chosen on would then shape the weights it is chosen for.
• Every regularizer and constraint Keras has. A penalty is L1, L2 or both, and a bound is a largest norm for each unit. An orthogonal regularizer, a unit norm, a non-negative constraint and a penalty on what a layer makes are refused by name where a Keras file says one, rather than read as something they are not.
• A norm clipped tensor by tensor. The norm of all the gradients together is PyTorch's clip_grad_norm_, and each value can be clipped on its own; nothing else says a clip.
• A start in a file. A file holds the numbers a layer learned, never how they began.
• A second word for a loss that is another's over a number. PyTorch's SmoothL1Loss is the Huber loss over its beta: say huber with its delta at that beta, which learns the same network with a loss beta times as large.
• A group or an instance normalisation in a declared network. Each measures over the places an image holds, and a pipeline hands a network a flat row of one place; for images, write the network as code or in Keras's words.
• A word of your own for a loss or an optimizer in a pipeline's file. Yours is code: compile and fit it by hand.
• Hand-written GPU code. A graphics card is reached through an engine that already reaches one: libtorch, through the TorchSharp backend.