Getting started

In short

This page installs DeepSharp, adds two tensors, works out a gradient, and then runs the whole course at once: a file read, prepared, and a network trained on it. The one rule to remember: everything below the split is fitted on the training rows alone and replayed unchanged on the rest.

You want to…	Write
install the core, the pipelines, the networks and the charts	dotnet add package DeepSharp (and .Pipelines, .Learners.Networks, .Charts)
show a notebook in your browser	deepsharp-serve Samples/titanic.verso
add two tensors	new CpuBackend().Add(a, b)
work out a gradient	new RecordingBackend(maths).GradientsOf(loss, [weights, bias])
prepare the rows	Pdd.Create().ReadCsv(…).Declare(…).SplitStratified(…).Target(…) … .Build().Run()
take the training rows out	prepared.Batch(Part.Train)
ask what each column holds	Pdd.Create().ReadCsv("titanic.csv").ProposedKinds()
train the network a pipeline names	pipeline.Train()
answer a row nobody trained on	trained.Predict(rows)
keep it as one file, and read it back	trained.ToJson() · TrainedNetwork.FromJson(text, NetworkCatalog.BuiltIn(), StepCatalog.BuiltIn().WithNetworks())
train on libtorch instead	pipeline.Train(new Engines().Use("torch", TorchBackend.OnCpu()))

The rest of this page explains each one, in that order.

Install

dotnet add package DeepSharp
dotnet add package DeepSharp.Pipelines
dotnet add package DeepSharp.Learners.Networks
dotnet add package DeepSharp.Charts

Which package does what

Package	What it brings
DeepSharp.Learners.Networks	a network trained behind a pipeline
DeepSharp.Charts	its charts
DeepSharp.Pipelines	the data half
DeepSharp.Pipelines.Parquet, .Excel, .Json	reading those files
DeepSharp.Pipelines.DataFrame	reading through Microsoft's DataFrame
DeepSharp.Pipelines.Indicators	indicators over a series
DeepSharp.Backends.TorchSharp	the arithmetic on libtorch — TorchSharp backend
DeepSharp.Import.PyTorch, .Keras, .Onnx	reading a model trained elsewhere — Importing a model

Add the ones you use.

A notebook

To write a pipeline as a notebook, install DeepSharp.Verso.Notebooks from Verso's Extensions panel. It runs in Verso's VS Code extension, in verso serve, and in an application of your own through DeepSharp.Verso.Api: see Notebook.

Or let DeepSharp's own server show it in your browser, with nothing else to install. Run it beside a notebook, or in a folder of them; from the repository's folder, this opens the sample notebook:

dotnet tool install --global DeepSharp.Verso.Serve
deepsharp-serve Samples/titanic.verso

From source

Or build it all from source:

git clone https://github.com/xkqg/DeepSharp.git
cd DeepSharp
dotnet build DeepSharp.slnx -c Release

Where it runs

It runs on .NET 8 and .NET 10, on Windows, Linux and macOS. The default engine needs nothing installed. libtorch, the heavier one, is a package of its own, and choosing it is one line.

Adding two tensors

using DeepSharp.Tensors;

var maths = new CpuBackend();

var a = Tensor.From(new Shape(2, 2), [1f, 2f, 3f, 4f]);
var b = Tensor.From(new Shape(2, 2), [10f, 20f, 30f, 40f]);

var sum = maths.Add(a, b);            // 11, 22, 33, 44
var product = maths.Multiply(a, b);

Console.WriteLine(sum);               // Tensor 2x2

Working out a gradient

A network learns by knowing which way to move each of its numbers. Hand the pass a RecordingBackend instead of the plain one, work out a loss of one value, and ask which way each number moved it:

var pass = new RecordingBackend(maths);

var rows = Tensor.From(new Shape(2, 2), [0.80f, -0.57f, 0.79f, -0.67f]);
var weights = Tensor.From(new Shape(2, 1), [0.1f, -0.2f]);
var bias = Tensor.From(new Shape(1), [0f]);
var wanted = Tensor.From(new Shape(2, 1), [0f, 0.01f]);

var misses = pass.Subtract(pass.AddRow(pass.MatMul(rows, weights), bias), wanted);
var loss = pass.Mean(pass.Multiply(misses, misses));

var gradients = pass.GradientsOf(loss, [weights, bias]);
Console.WriteLine(gradients[weights]); // Tensor 2x1: how much each weight moves the loss

The recording is made for one pass and dropped after it; nothing else holds it.

Two ideas the rest is built on

A wrong size is caught where you write it

new Shape(2, 3) means two rows of three. Add a 2x3 to a 3x2 and you get an error naming both, not a number that quietly went wrong three layers later.

A shape with no sizes at all is a single number. That is what a loss is, and it behaves like one everywhere.

You hand over the maths; nothing goes looking for it

There is no DeepSharp.Backend.Current and there never will be. You create a backend and pass it to what needs it. A shared one that anything could reach and replace is how two halves of a program end up disagreeing about where their work runs.

It is also the whole point of the design: your model talks to a backend, so pointing it at libtorch instead of at the built-in engine is one line, and the model itself does not change:

• to train on it: new FitOptions(seed: 42) { Backend = TorchBackend.OnCpu() };
• to serve on it: trained.Predict(rows, engine).

A tensor never changes

Tensor.From copies the numbers you give it. So if you reuse your own array afterwards, the tensor you already handed over stays as it was, and giving the same tensor to two layers is safe. Nothing writes into what you passed in.

The course, step by step

The ten steps below are the course this library is named after, one page each: what the step is for, the one line you write, what the row looks like after it, and where every option lives. Walked in order they build one pipeline from a file to a trained model; read on their own they answer "where does this belong?".

	step	what it decides
1	Reading a file	where the rows come from, and what each column looks like before you decide
2	Declaring the columns	what takes part, what kind each column holds, and which may be absent
3	Settling the gaps	the gaps a feature will be worked out from, with a value nobody learned
4	Working out the features	the columns worked out from other columns, above the split so the split divides them
5	Splitting the rows	the line: which rows may teach, which may choose, which may only measure
6	Naming the answer	what the model is asked to predict, kept apart from what it is given, named right below the line
7	Filling and scaling	everything learned from the training rows and replayed unchanged
8	Measuring and drawing	how every run is measured, declared before a number exists
9	Naming the network	which network these rows are prepared for, in Keras's words or PyTorch's
10	Saving and replaying	the one file, and what a replay is held to

To start a pipeline from these steps rather than from memory — every one present, in this order, waiting for what only you know — see Starting from a course.

The rest of this page is the short version: the whole course run at once.

Preparing real data

using DeepSharp.Pipelines;

var passengers = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "pclass").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .Normalise("age", "fare")
    .Build()
    .Run();

var train = passengers.Batch(Part.Train);   // the numbers, their names, and the answers apart

Everything below the split is fitted on the training rows alone and replayed unchanged on the rest. PDD says why, and Pipeline has every verb.

One line, many columns

One line does one thing to many columns, and each writes one step a column, as writing the verb once a column does.

• .Normalise("age", "fare") scales both the way a caller who names no kind means it: between minus one and one, which is where a network takes its features.
• .Normalise(scale => scale.MidRange("age", "fare").Robust("volume")) names a kind per group.
• .FillMissing(fill => fill.Median("age").Mean("trades")) is the same shape for the gaps.
• .Reshape, .Cyclical, .ClipOutliers, .FillNaN and .TimeParts take the same shape too.

What stands above the split

Some lines stand above the split, because nothing in them is learned from the rows:

• .SettleGaps(gaps => gaps.Zero("trades")) has the same shape. A nought, a number you choose or a refusal is not a value any row decided, so a feature worked out after it is worked out from settled columns.
• .ScaleGiven(scale => scale.Between("fare", 0, 512)) stands there for the same reason: the bounds are yours rather than the rows'.
• .Shuffle(seed) puts the rows in an order drawn from a seed before they are divided, so every part holds the same mixture. Without it they keep the order they were read in.
• .DefaultFeatures(Form.Unit) says, once for the whole pipeline, where the features land: between nothing and one instead. A column that names its own kind keeps it.

Not sure what a column holds?

Pdd.Create().ReadCsv("titanic.csv").ProposedKinds() reads every cell and proposes a kind for each column — alive true or false, embarked a category, age a number — to write the schema from. It is a proposal and nothing more: the schema is still yours to write.

Training a network on it

A network takes every feature declared to land between minus one and one, so the scales say so. The pipeline declares what the network will be held to before anything learns, and then the network itself: the layers, what moves them, what judges them and when the run stops. Running it trains exactly that network:

using DeepSharp.Charts;
using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")      // a network takes them between -1 and 1, which is the default
    .Report(report => report.Measure(Metric.Accuracy, Metric.ConfusionMatrix).On(Part.Validation, Part.Test).As(Shown.Numbers))
    .WithTensorflow(network => network                         // the network these rows train
        .Dense(16).Relu().Dense(1)                             // the widths come from the rows
        .Adam(0.01)
        .BinaryCrossEntropy()
        .Run(seed: 20260929, epochs: 100)
        .StoppingAfter(patience: 10))
    .Build();

var trained = pipeline.Train();                                // the steps, and then the network they name

var chance = trained.Predict(new InMemoryRowSource(
    ["pclass", "sex", "age", "sibsp", "parch", "fare"], ["3", "male", "22", "1", "0", "7.25"](/xkqg/DeepSharp/wiki/"3",-"male",-"22",-"1",-"0",-"7.25")));

File.WriteAllText("titanic.network.json", trained.ToJson());    // the network and its pipeline, one file
File.WriteAllText("titanic-loss.svg", trained.History!.LossCurve());

What the run does

The whole course is one declaration, so the file says what was trained behind its steps, and training it again from the file gives the same network.

• It learns from the training rows.
• It is judged by the validation rows, and stops once they no longer improve.
• The test rows reach it only when the report measures it, each measure beside what predicting the training rows' average would score.

Predict answers a passenger nobody trained on with the chance they survived.

Reading it back

The file reads back with TrainedNetwork.FromJson(text, NetworkCatalog.BuiltIn(), StepCatalog.BuiltIn().WithNetworks()), beside the very pipeline fit it was trained behind and no other. The catalog is told the verb the declaration names.

The same network written as code

The same network written as code trains the same, bit for bit: new Sequential().Dense(16).Relu().Dense(1).Compile(new Adam(0.01), new BinaryCrossEntropy()).Fit(prepared, options), over a pipeline that was only .Run(). That is the door to take when the network is not yours to write — one read from a Keras or PyTorch file, say.

Choosing the engine

The run happens on the engine the declaration names, light unless said. An application gives that name an engine before the run: pipeline.Train(new Engines().Use("torch", TorchBackend.OnCpu())). trained.Predict(rows, engine) serves on whichever engine it is handed. Networks has all of it.

What else a layer and a run can be told

A layer can also be left without its bias, started another way, penalised, bounded or held still. A run can clip each gradient value, take a step from several batches and keep an average of its weights. Each is off until asked, and Networks says which to reach for when.

Running the samples

Two programs and a notebook in the repository run all of this on the published files in Samples/data:

dotnet run --project Samples/DeepSharp.Sample.Pipelines -c Release -f net10.0
dotnet run --project Samples/DeepSharp.Sample.Networks -c Release -f net10.0

Sample	What it does
DeepSharp.Sample.Pipelines	asks the passenger list what each of its columns holds before anything is declared, has a profile say what should not be there and how each is answered, and prepares the passengers and a price series, reaching its pipelines the way an application reaches a service
DeepSharp.Sample.Networks	trains a network on each of three datasets, has each pipeline's report measure it, saves it as its one file, reads it back and serves a row, and writes its charts beside itself
Samples/titanic.verso	writes the Titanic pipeline block by block; its C# cell trains a network on what the blocks hand over, and its report block draws the measures — open it in Verso, or with deepsharp-serve as above

A test runs each as it stands.

Running the tests

A test project is a program you run, not a suite you point a tool at, so dotnet test is not how it works here. There are six:

• the library's;
• the libtorch engine's, which carries libtorch, as the library's never does;
• the ML.NET learner's, which carries ML.NET and the native side it comes with;
• the notebook's, the host's and the server's, which run on Verso's own engine.

Each is built for .NET 8 and for .NET 10, so -f says which to run on; -f net8.0 runs the same tests on .NET 8:

dotnet run --project Tst/DeepSharp/DeepSharp.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp.Backends.TorchSharp/DeepSharp.Backends.TorchSharp.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp.Learners.MLNet/DeepSharp.Learners.MLNet.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp.Verso.Notebooks/DeepSharp.Verso.Notebooks.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp.Verso.Api/DeepSharp.Verso.Api.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp.Verso.Serve/DeepSharp.Verso.Serve.Tests.csproj -c Release -f net10.0
dotnet run --project Tst/DeepSharp/DeepSharp.Tests.csproj -c Release -f net10.0 -- -class DeepSharp.Tests.Tensors.ShapeTests

The library's suite compiles every C# block of this wiki against the packages as built, and the notebook's runs every block marked as a notebook's cell in Verso's own engine. So both need the wiki cloned beside the repository, as DeepSharp.wiki.

To see how much of the library the tests actually reach — which is a check that fails, not a number to admire — see Quality.