The pipeline

This page is the vocabulary: every verb, in the order you write it. All of it runs today — reading a file, proposing what each column holds, declaring its columns, putting the rows in order, working out features and indicators, dividing the rows three ways, what a model is asked to predict — one column, the shares of a whole, labels, or an answer read from later rows — filling gaps, encoding categories, seven scales, the evidence a run produces, and the handover to whatever learns, run for what that learner needs — each learning step fitted on the training rows alone and replayed unchanged everywhere after, with what it learned written into the file beside the declaration — and the report of what a trained model is measured by. The first learner, a network, is built too, and Networks has it, and a declaration can name the network its rows train; a trainer from ML.NET is built too, and ML.NET learner has it.

Why it is built this way at all is PDD. The same pipeline written block by block, with the data under any block, is the Notebook. And the reference of every verb exactly as a file holds it — VERBS.md — is written by the steps themselves, so it cannot drift from them.

The verbs live in DeepSharp.Pipelines, which is its own package: it knows nothing about tensors, and a service that only prepares data never carries an engine it does not call. The indicators, the readers of a Parquet file, an Excel workbook and a JSON file, and the data-frame reader are packages of their own too.

The whole thing in one look

using DeepSharp.Pipelines;

var passengers = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema
        .Integer("survived", "pclass", "sibsp", "parch")
        .Number("fare")
        .Optional("age", ColumnKind.Number)
        .Category("sex", "embarked"))
    .AddFeature("family", "sibsp", Arithmetic.Plus, "parch")
    .SplitStratified("survived", train: 0.70, validation: 0.15)   // test is the rest
    // ---- nothing above this line is allowed to learn from the data ----
    .Target("survived")                                           // the answer, named right below the line
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "family")
    .Build()
    .Run();

var train = passengers.Batch(Part.Train);   // rows of numbers, their names, and the answers apart

Everything is reached through a factory and built as a chain, and the chain is not decoration: the stages are different types, so the steps that learn from the data are not offered until you have said how to split.

The chain is where you meet that rule; it is not where the rule lives. A step that learns says so in its own type, and the check is made on the finished declaration — so it holds whether the step came from the chain, from a verb another package added, out of a file somebody edited by hand, or from a notebook.

The rules every pipeline keeps

Whichever way a pipeline was written, the declaration refuses it unless:

• it has at most one source, one schema, one split, one order and one output — a second one used to be ignored, so a file said one thing and the numbers came from another;
• the source is the first step, and the schema comes directly after it, because everything else works on columns;
• nothing that learns from the data stands above the split;
• rows are dropped and put in order above the split, never below it, because the split divides the rows it is given once;
• a step that reads the rows in their order stands below the step that says what that order is;
• every column a step reads is there where it reads it, of a kind it can work on;
• only an output reads rows after its own, below a split in time whose gap is at least as wide as how far it reads, with the rows ordered by that split's column alone;
• every step does something the run acts on.

A refusal names every fault at once, each with the step it is at — DeclarationException.Faults — and PipelineDeclaration.FaultsIn(steps) gives the same list without refusing, for something that writes a pipeline a piece at a time.

Reading the data

One verb per source, each an extension method living in the package that brings what it needs:

.ReadCsv(path)	DeepSharp.Pipelines	A comma-separated file.
.ReadParquet(path)	DeepSharp.Pipelines.Parquet	An Apache Parquet file, which says what each of its columns holds.
.ReadExcel(path), .ReadExcel(path, sheet)	DeepSharp.Pipelines.Excel	A sheet of an .xlsx, .xls or .xlsb workbook: the first, unless one is named.
.ReadJson(path)	DeepSharp.Pipelines.Json	A JSON file holding an array of records, one object a row, as an API hands them back.
.Read(rows, description)	DeepSharp.Pipelines	Rows handed in: anything that can be turned into an IRowSource. The file records what they were, rather than pretending it can open them again.
.ReadDataFrame(frame)	DeepSharp.Pipelines.DataFrame	A data frame you already hold: Microsoft's DataFrame, from Microsoft.Data.Analysis.
.ReadCsvFrame(path)	DeepSharp.Pipelines.DataFrame	A CSV read through the frame's own reader, every column as text, as the file writes it: the same rows ReadCsv reads.
.ReadDbAsync(reader)	DeepSharp.Pipelines.DataFrame	The rows of a database query, through any ADO.NET provider, with what the database says each column holds.
await ….ReadBinanceAsync(candles)	DeepSharp.Pipelines.Binance	The candles of a market on Binance, landed as a file in the working directory where they are not landed yet, and read as that file: an ordinary .ReadCsv of it. See Live sources.
.ReadJoin(left, right, on, unmatched)	DeepSharp.Pipelines	Two comma-separated files as one source: each row of the left one beside the row of the right one its key names.

Reading is a solved problem with a long tail — pandas exposes nineteen readers — and reproducing that tail is a library in itself. Anything else is one small IRowSource away.

A reader of a file is a verb of the pipeline's file as well — read.parquet, read.excel and read.json beside read.csv — opened when the pipeline runs and read again whenever it is, so a pipeline file written with any of them replays. A catalog reads them once its package teaches it: StepCatalog.BuiltIn().WithParquet().WithExcel().WithJson(), and a file that names one a catalog was never taught is refused with the name of the package it needs.

Every value reaches the pipeline as text, and a row is known by that text, so each reader hands a cell over as its file holds it. JSON is text, and a number keeps its spelling there: 22.0 stays 22.0. A Parquet file and a workbook hold values rather than text, and hand them over by one rule the database reader shares: a number in its shortest exact form, a moment or a day as ISO 8601 writes it, true and false as True and False. So the Titanic passenger list, written in each format with the cells its comma-separated file holds, gives the same keys, the same split and the same batches in all four. Each format has a few rules of its own:

• A workbook is read from its first sheet unless the pipeline names another — one it does not have is refused with the names of those it has — and the sheet's first row names its columns. A cell is read as the sheet types it, never as the text a culture shows it as, a date as a moment; an empty cell is empty text, as the same sheet saved as comma-separated text holds it, and a formula's error is spelled as Excel spells it there. The library it reads with asks .NET for a Windows code page before it opens any workbook, so the package offers .NET's own code pages, once for the whole program, and every encoding the program had stays as it was.
• A JSON file is an array of records: its columns are the keys in the order they first appear, a key a record leaves out and a null are both gaps, and a value that is itself an object or a list is refused by name, with its record.
• A Parquet file is read row group by row group in the order it holds them, and a column of lists, of groups of fields or of raw bytes is refused by name rather than flattened by a guess.

A relative path is read from the folder the pipeline sits in: the folder of the file it was read from, or of the notebook it was written in, and the working directory for a pipeline written in code. So a pipeline file moved together with its data still finds it. Hand the folder over with new Pipeline(declaration, rows, folder) and SourceFolder.OfDocument(path).

A live source is fetched, then read. An endpoint answers differently every time it is called, so a pipeline that read from one during training would train on different numbers tomorrow while claiming to be the same pipeline. A fetch lands the result as a file, recording what it asked for and what came back, and the pipeline reads that file like any other: DeepSharp.Pipelines.Binance does it for the candles of a market on Binance, and the pipeline's file names the landed file and nothing of Binance. See Live sources.

Two files, one source. .ReadJoin("planned.csv", "arrived.csv", ["flock"], Unmatched.Drop) reads two comma-separated files, each as .ReadCsv reads one, and gives the rows of the left one, in its order, each beside the row of the right one its key names. The key is the exact text of the on columns with the spaces around it taken off — no case is folded and no number is read, so 07 is not 7 — and a key cell that is empty matches nothing. The join is many to one: several left rows may share a key and each takes the same partner, but a key the right file holds twice is refused, naming it and its rows. A column both files have beside the keys is refused by name. What becomes of a left row with no partner is always said, and has no default: Unmatched.Refuse stops the run naming the row and its key, and Unmatched.Drop leaves the row out and the split writes how many as rows.unmatched. Rows served later arrive already joined, since rows handed in take the place of the source, and a file with a row for each band is pivoted before it is joined and not by the join. In a file the verb is read.join, new in the eighth version; a notebook shows the joined rows and keeps them by a fingerprint of both files.

Declaring the columns

.Declare(...) comes directly after the source. It names the columns that take part and what each holds — Text, Number, Integer, Boolean, Timestamp, Category — marks the ones that may be absent with Optional, and says what becomes of the rest: dropped by default, because a column nobody declared is a column nobody checked. Remainder.Keep carries them along as text, for a pipeline still finding out what is in the file, and Remainder.Refuse stops the run when the file holds one.

A declared column can be left out without being forgotten: "excluded": true keeps it in the schema with its kind, while nothing reads it and the source is not asked for it, and a step that reads it is refused at that step as one "which the schema excludes". A category can say which kind it was before it became one, so it can go back. Both are written only where they say something, so a schema that uses neither is written as before. A file that uses either is for 0.3 on: 0.2 would read an excluded column as one that takes part.

One column can be the id: schema.Id("flock", ColumnKind.Integer).Number("age", "weight") marks the number a team keeps beside its rows — whole numbers, categories or text, at most one, and never the answer. It is carried with every row as it was read and is nothing else: not a feature, not encoded, and left out of a profile or a warm-up that is not told its columns. It is handed over as Ids on each Batch and ServedBatch and on the predictions of a trained network, so an answer finds its row by what the row says and not by where it stood. In a file it is "id": true on its column, written only where it is true, and 0.8.0 refuses the key by name.

A timestamp column can say the format its moments are written in, as .NET writes a date format: .Column(new ColumnDeclaration("when", ColumnKind.Timestamp, Optional: false) { Format = "dd/MM/yyyy" }) in the chain, "format": "dd/MM/yyyy" on the column in the file. Without one, its moments are read as ISO 8601 writes them — 2015-02-18, 2015-02-18 09:30, 2015-02-18T09:30:15, with a fraction of a second, a zone or an offset or without — and any other writing is refused. A column with a format still reads the round-trip form a database or a data frame hands its moments over in. Nothing is guessed: a reading that tried every form it knew read 7.25 as the twenty-fifth of July of the year it ran in, and 02/03/2015 as the third of February whoever wrote the file.

Any column can say which of its values stands for a gap: missing, 0 for a fare nobody wrote down — .Column(new ColumnDeclaration("fare", ColumnKind.Number, Optional: false) { Missing = "0" }), or "missing": "0" on the column in the file. Every cell that holds it is read as a gap, so fill.missing counts and marks it. It is compared as the column's kind reads it, so 0 and 0.0 are one value, or as it is written when it is not of that kind, so a column of numbers can say that ? is a gap.

A schema does not have to be typed from nothing. ProposedKinds(), on the chain before .Declare(...), reads every cell of every row and proposes what each column holds, with how many cells hold a value, how many are gaps and how many different values there are:

using DeepSharp.Pipelines;

var proposal = Pdd.Create().ReadCsv("titanic.csv").ProposedKinds();

var embarked = proposal["embarked"].Kind;   // Category: three different words, each standing for many rows
var pclass = proposal["pclass"].Kind;       // Integer, with Category offered beside it: a class and a count look alike
var alive = proposal["alive"].Kind;         // Boolean: yes and no

It is a value to write the schema from and nothing takes it in, so a pipeline never works its schema out again from whatever the file holds on the day it runs. Moments are proposed as ISO 8601 writes them, or with the one usual format that reads every cell; cells that read day first and month first alike are offered with both formats, for you to say which. A query read with ReadDbAsync says what its columns hold as the database types them, and so does a Parquet file, and what either says outranks what the cells look like: a code the database holds as words stays words.

Everything below is checked against it. The columns are followed from the schema down, each step saying what it leaves behind, so a step that reads a column the schema left out, or one a step above took away, is refused where it stands — not a whole run later, as a column nobody could find. When the file is opened, the declared columns are checked against the ones really there before the first step runs.

Values are read the same way on every machine: numbers under the invariant culture, moments as ISO 8601 or their column's format writes them, true and false in the spellings files actually use — True and False included. A cell that cannot be read as its column's kind is refused with its row, its column and its value, and a number too large to hold, such as 1e999, is refused rather than read as infinity. Every column with such cells is named at once, at the schema's step: a column with a few is named cell by cell, one with more has its first three named and the rest counted.

Putting the rows in order

.OrderBy(columns) puts them in the order a series needs. .Shuffle(seed) is the other answer: it puts them in an order drawn from a seed, before they are divided, so every part holds the same mixture rather than the one the file happened to be written in. Without either, the rows keep the order they were read in. They are never written together — a pipeline cannot be told two orders — and a series in time wants .OrderBy, never a shuffle. The order is drawn the way the splits draw theirs, by a digest of what each row says together with the seed, so the same rows give the same order on every machine; the training loop shuffles again within each epoch, from its own seed, which is a different question.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.High", "AAPL.Low", "AAPL.Close"))
    .OrderBy("Date")                                          // oldest first
    .Add(new AddIndicatorStep("rsi", Indicator.Rsi, ["AAPL.Close"], 14))
    .DropWarmUp()
    .SplitByTime("Date", train: 0.70, validation: 0.15)
    .Build()
    .Run();

A file does not promise an order. An export or a query without an ordering hands rows over in whatever order it happened to hold them, and a step that reads the rows before a row then reads the wrong ones: the same prices listed newest first gave a five-day average of 98.352 where the right one is 99.74. So a step that reads the order — an indicator, DropWarmUp, a fill that carries the previous value forward — needs an OrderBy above it, and is refused without one.

OrderBy takes one or more columns and puts the smallest first, comparing the values in their own kind. It refuses a gap in a key, and two rows whose keys are equal: nothing then says which came first, so add a second key column.

Features

A feature is worked out from a single row — or, for an indicator, from the rows that came before it — and learns nothing from the data as a whole, so features stand above the split.

.AddFeature(name, left, Arithmetic.Plus, right)	A column from two others: Plus, Minus, Times, DividedBy.
.Cyclical(column, Period.HourOfDay, form)	A moment as a place on a circle: HourOfDay, DayOfWeek, DayOfMonth, MonthOfYear, DayOfYear over the length of its own year, and Season. Many at once: .Cyclical(cycle => cycle.HourOfDay("start").DayOfWeek("start")), written where the pipeline says its features land.
.Cycle(column, length, form)	A number as a place on a circle of a length you declare — an age in days around a week, .Cycle("age", 7). Many at once: .Cycle(cycle => cycle.Every(7).Of("age", "day").Every(30).Of("phase")). The length is said and never taken from the data's largest value. In a file it is feature.cycle, new in the eighth version.
.TimeParts(column, TimePart.Month, …)	The pieces people reason with — Minute, Hour, DayOfWeek, DayOfMonth, Month, Quarter, Season, Year — as categories. Many at once: .TimeParts(parts => parts.Taking(TimePart.Month, TimePart.Hour).Of("start", "end")).
.TimePartsAsNumbers(column, …)	The same pieces as numbers, for the one where the order really is the point; the same line, under this name. A weekday and a season have no number to be, and are refused where they are written, naming Cyclical and categories.
.Reshape(column, Maths.Log)	Log, Log1P, Reciprocal, Sqrt, Square, ArcSin, Abs, Sign. Many at once: .Reshape(shape => shape.Log("fare", "volume").Sqrt("range")), with the column to write the result into left on the verb that takes one.
.Add(new AddIndicatorStep(name, Indicator.Rsi, columns, period))	An indicator, from DeepSharp.Pipelines.Indicators.

Cyclical values. An hour of day is not a number between 0 and 23: 23:00 and 00:00 are neighbours, and a model given the plain number is told they are as far apart as possible. The angle is written as a sine and a cosine, which puts the wrap where it belongs, in one of three forms:

Form.Signed	one column each, -1..1	the plain value
Form.Unit	one column each, (x+1)/2 into 0..1	everything on one range; zero is now 0.5
Form.SplitSign	two columns each, 0..1	pos = max(x,0), neg = max(-x,0)

Signed and Unit are the same feature up to a shift, so a layer with a bias absorbs the difference. SplitSign adds something: each half gets its own weight, so a rise and a fall can be answered differently. It has to be the last form a value takes — normalising one half by its own extremes would give one quantity two slopes — so a scale on a half is refused.

The parts of a moment are categories, because a month is not a quantity: March is not three of anything.

Indicators — Sma, Ema, Rsi, Atr, Adx, Cci, WilliamsR, Obv, Macd, BollingerBands, Stochastic, Vwap — are borrowed from MatPlotLibNet rather than written again. Two things about them are measured rather than promised: each is held to an impulse test — change one row, and no earlier row may move — and the warm-up arrives as a gap, because an indicator of period N has nothing to say about the first N rows. DropWarmUp() drops the rows at the start that no column can speak for yet: 506 Apple rows with a twenty-day band on them are 487 rows of data.

Taking something away

.Drop(columns)	Leaves columns out from here on — above the split or below it.
.DropGaps(columns)	Drops every row with a gap in one of the named columns, above the split.
.SettleGaps(column, with)	Settles the gaps with a value no row decided, above the split, where the features are worked out.
.DropWarmUp()	Drops the rows at the start that an indicator cannot yet speak for, above the split.

Rows are dropped before they are divided, never after: dropping rows below the split would quietly change how many each part holds, which is what the split promised.

The split

.SplitByTime(...), .SplitAtRandom(...) and .SplitStratified(...) are the line in the chain. Above it nothing may learn from the data; below it the steps that do become available. Which one to use is not the library's decision — by time when you predict forward, at random for rows that do not depend on one another, stratified when an answer is rare — so none of them is the default.

A split divides rows by what they say, not by where they stand. Each row is ranked by a digest of its own contents and the seed, so the same rows are dealt the same way whatever order the file lists them in, and every copy of a repeated row lands where its first copy does. A split in time never divides a moment: every row of one moment lands on the side of the line its first row does.

The test rows can be dealt apart from the rest. A seed deals every part, so a second seed also deals another test part, and a search that tries many seeds would choose on rows another seed learned from. .SplitAtRandom(0.70, 0.15, seed: 7, testSeed: 99) deals the rows to be measured on, and the rows kept to predict on, by testSeed alone, so they are the same whatever seed is, and seed deals what is left into training and validation. Left out, it is not written and seed deals everything; equal to seed, it deals exactly as no test seed does.

A row does not belong to a split, it belongs to a part: you split the rows, and Part.Train, Part.Validation, Part.Test and Part.Predict are what they land in.

The number you do not write

You say what training takes and what validation takes. The share you are measured on is whatever is left — it is not a parameter, so nothing can add up to more than there is.

using DeepSharp.Pipelines;

var byTime = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .SplitByTime("Date", train: 80, validation: 10)          // test: 10
    .Build();

var twoWays = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitAtRandom(0.80)                                      // 0.80 and 0.20, no validation
    .Build();

Percentages and fractions say the same thing — 80, 10 and 0.80, 0.10 — and mixing them in one call is refused rather than read as eight-tenths of a percent. Three numbers that have to add to one is a rule that breaks quietly: 0.70, 0.15, 0.10 leaves a twentieth of the rows in no part at all.

A part that takes no part

Predict(10) holds a tenth out of everything. Nothing is fitted on it and nothing is measured on it, so running a trained network over those rows is the closest thing to running it tomorrow; with a split in time they are the newest days in the file.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .Predict(10)                                        // held out of everything
    .SplitByTime("Date", train: 65, validation: 15)     // test: 10 — the four make a whole
    .Drop("Date")                                       // a moment is not a number a model takes
    .Normalise("AAPL.Close", Scale.Robust)
    .Build()
    .Run();

var later = prices.Batch(Part.Predict);                 // handed over like any other part

It is written once, before the split, and the split is what sets it aside — so a pipeline that holds rows back and then never divides anything is refused where it is built.

A gap between the parts

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .OrderBy("Date")
    .SplitByTime("Date", train: 0.70, validation: 0.15, gap: 5)   // five days kept apart at every line
    .Ahead("AAPL.Close", 5, AheadAs.Return)                        // the return five days on
    .Drop("Date")
    .Build()
    .Run();

A split in time can keep a gap: the last moments of every part — before each line, and at the end — are set apart as Part.Gap, fitted on by nothing and handed to nothing. It is there for an answer read from later rows. Without it, the last training days of an answer five days ahead read their answers from the days a model is measured on. The gap counts moments rather than rows, so it never divides a moment, and a file that keeps no gap is written exactly as before.

What a model is asked to predict

The answer is named right below the split, before anything that learns, and there are five kinds of it:

.Target(column)	target	One column.
.Distribution(columns, scaleBy)	target.distribution	The columns a whole is divided among, in their order: at least nought and summing to one on every row. scaleBy, the column saying how many there were, is none of the shares. ordered: true says the order means something, as bands of weight do, and remainder names one more answer for what is left of the whole.
.Numbers(columns)	target.numbers	Two columns or more that are each any finite number: amounts that are not shares of a whole and not classes.
.Labels(columns, ones)	target.labels	Columns that are each nought or one, with how many ones a row holds when you say so — never more than there are labels.
.Ahead(column, ahead, as)	target.ahead	A column rows later in the declared order: the value then, or the return on the row's own value.

using DeepSharp.Pipelines;

string[] bands = [.. Enumerable.Range(0, 70).Select(band => $"w{500 + (50 * band)}")];   // 500 g to 4000 g

var flocks = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema.Timestamp("weighed").Number("age", "chicks").Number(bands))
    .OrderBy("weighed")
    .SplitByTime("weighed", train: 0.70, validation: 0.15)
    .Distribution(bands, scaleBy: "chicks", ordered: true)
    .Drop("weighed")
    .NormaliseRow(Norm.L1, bands)          // counts become shares: each row sums to one
    .Build()
    .Run();

A flock weighed in seventy bands of fifty grams is one answer of seventy numbers. The shares are checked on every row as they are handed over — one below nought, or a row that does not sum to one, is refused, naming normalise.row. Named with the column saying how many birds there were, predicted shares come back as birds in each band, by the flock's own size; and the run checks, on every training row, that the bands add up to it.

Bands in an order say so, with ordered: true: then Metric.Emd, Metric.Kl and Metric.Rps measure the shares along it and the loss earthMoversDistance trains on them — see Networks. A key that is false unless said is no guess of the library's about the data, and the measures that follow an order are refused against an output that does not say it has one. A flock that arrives with fewer birds than were planned does not add up to the planned flock, and remainder: "other" makes one more answer, after the bands, of what is left of it. The bands then hold counts where the output stands, so no normalise.row stands above it; the shares sum to one and come back as birds that add up to the planned flock, and more birds than were planned is refused, naming the row. In a file ordered and remainder are optional keys of the eighth version. .Numbers(["weight", "age"]) is the answer of several free numbers; it comes back in its own units, as .Target's does.

target.ahead makes its answer where it stands, when the pipeline is fitted, from the rows in their declared order; the last rows, with nothing that far after them, have none. A served row has no later rows, so it awaits nothing from the rows handed in. A return comes back as a price by the day's own price as it was read, which is why a return stands above every step that changes the column it is made from.

Gaps

Two things usually thrown onto one heap are kept apart. A value is missing when it was never there — an empty field, a NULL — and that is data. A value is not a number when arithmetic produced none, which is almost always a fault further upstream.

using DeepSharp.Pipelines;

var filled = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .FillMissing("age", With.Median, refuseAbove: 0.5)   // With.Mean, With.Zero, With.Constant(0),
    .Build()                                             // With.Previous, With.Refuse
    .Run();

The strategy is a named value rather than a flag, because Fill("trades", true, false) tells the next reader nothing. Whatever fills the gap is learned, from the training rows alone, and so is what the fill counts: the Titanic ages have 177 gaps in the file and 133 among the rows a stratified split trains on.

A column is written alongside, always, saying where the gaps were — age_was_missing. Filling destroys the difference between "absent" and "the value happened to be that", and it destroys it for good; if you do not want the column, you drop it.

A fill has a point beyond which it is invention. deck is empty in 688 of the 891 rows, and filling a column like it would manufacture three quarters of it. refuseAbove is the share of the training rows that may be gaps and still be filled. Above it the column is not filled, and the column that marks the gaps speaks for it. It has no default, because a number nobody chose would be the same quiet decision somewhere else — which is also why it stays on the verb that takes one column rather than on the line below.

One line fills many columns, the value named per group: .FillMissing(fill => fill.Median("age").Mean("trades")), and .FillNaN(fill => fill.Refuse("range").Zero("ratio")) for what arithmetic could not make. Both choose from the same list — Mean, Median, Zero, Previous, Refuse, Constant(value) — which is written once, because they are two different questions with one answer each.

A gap a feature is worked out from is settled above the split, not filled below it. A fill learns its value from the training rows, so it stands below the split — and the features are added above it, which means a column derived from one with a gap is itself a gap, and filling that column afterwards does not reach back into it. .SettleGaps is the other half of the same work: a nought, a number you choose, or a refusal, none of them a value any row decided, so it may stand where the features are.

using DeepSharp.Pipelines;

var settled = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp").Optional("age", ColumnKind.Number))
    .SettleGaps(gaps => gaps.Zero("age"))                    // Zero, Constant(value), Refuse
    .AddFeature("older", "age", Arithmetic.Plus, "sibsp")    // worked out from a settled column
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("age", "older", "sibsp")
    .Build()
    .Run();

It writes nothing into the fitted half of the file, because there is nothing it learned, and it marks where the gaps were exactly as a fill does. The ways a fill learns — Mean, Median — are not among its kinds, and neither is Previous, which reads the rows in their order: each of those is decided by rows, and belongs below the split.

Settle a column or fill it, not both. Settling leaves no gap behind, so a .FillMissing of the same column further down can do nothing at all — and the pipeline says so where you wrote it rather than running a line that does nothing:

Step 6, 'fill.missing': would fill the gaps in 'age', and settle.gaps at step 3 already settled them: there is no
gap left for it to find. Keep the settling, or keep the fill and settle another column.

The column that says where the gaps were is the only record that a value was ever absent, since filling destroys the difference on purpose. So whichever verb marks a column first is the one that keeps it: a later marking adds to that answer and never replaces it.

.FillNaN(column) deals with a value that is not a finite number, and refusing it is the default: a not-a-number or an infinity is somebody's broken division, and carrying it into a model as if it were a measurement is the one thing a pipeline should not do quietly. Every fit refuses one among its training values, naming fill.nan, and the handover refuses one too.

Categories

.Encode(column, As.OneHot) learns which categories exist — from the training rows, like everything else below the line. As.Ordinal writes one column of places instead. An unfamiliar category turns up in production sooner or later, so what happens then is part of the declaration: Unseen.Reserve keeps a slot for it, Unseen.Refuse says plainly that the data is outside what the model was trained on.

Which columns are categories is said where the data is declared — Category("sex", "embarked") in the schema — and .EncodeCategories() then takes them all by name, as one step. A category that never became numbers is refused at the handover rather than dropped in silence.

One-hot with Unseen.Reserve, the place kept for an unfamiliar category is a column named after the encoded one and other, and the column that marks a gap one named after it and was_missing — sex_other, sex_was_missing. A category the training rows hold under either name would take that column and lose its rows' category without a word, so such a fit is refused, naming unseen: refuse and as: ordinal, which write no such column. A gap is no category: it sets no column one-hot, and takes no place — minus one — when the categories are handed to a learner that takes them as they are, below.

Normalising

Each kind learns something different, which is why the kind is declared and what it learned is stored apart:

	what it does	what it learns
Scale.Standard	(x - mean) / spread	mean and spread
Scale.MinMax	into 0..1	the smallest and largest value
Scale.MaxAbs	divide by the largest magnitude	that magnitude
Scale.Robust	median and the middle half of the data	median and quartiles
Scale.Quantile	rank, then spread evenly	the shape of the distribution
Scale.Power	reshape towards a bell curve	one parameter per column
Scale.MidRange	into -1..1, the middle of the training range at nothing	the smallest and largest value

Standard and min-max are both pulled around by a single extreme value, so one spike can squeeze everything else into a sliver of the range. On prices and volumes that is the normal case, which is what Robust and Quantile are for. MidRange lands the training rows between minus one and one, which is where a network takes its features — scikit-learn's MinMaxScaler with a feature range of minus one to one. The training rows' own extremes land exactly on its ends, rather than a rounding past them: the training maximum of a published price series otherwise came back as 1.0000000000000004, and a scale that refuses what lies outside refused the very fit it came from.

One line scales many columns, and the kind is the method. .Normalise("age", "fare") scales both the way a caller who names no kind means it, and .Normalise(scale => scale.MidRange("age", "fare").Robust("volume")) names a kind per group — one step a column either way, so the file, the notebook's blocks and every fit are what they always were. What happens outside the learned range is said on the kind that can hold a value in one: .MinMax(OutOfRange.Refuse, "age"). The kinds that land their rows in no range have no such method, and Quantile has only that one, so neither pair can be written wrongly.

Where the features land is said once, for the whole pipeline. .DefaultFeatures(Form.Signed) is between minus one and one, which is what a pipeline that says nothing means, and .DefaultFeatures(Form.Unit) is between nothing and one; a column that names its own kind keeps it.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("btceur-1d.csv")
    .Declare(schema => schema.Timestamp("timestamp").Number("close").Optional("trades", ColumnKind.Number))
    .OrderBy("timestamp")
    .DefaultFeatures(Form.Unit)                      // these features land between nothing and one
    .SplitByTime("timestamp", train: 0.70, validation: 0.15)
    .Target("close")
    .FillMissing(fill => fill.Mean("trades"))
    .Normalise(scale => scale
        .Columns("trades")                           // written as min-max, because that is where the features land
        .Robust("close"))                            // a column that names its kind keeps it
    .Build();

It is written above the split, where it holds for every column alike, and it says where a scaling that names no kind lands and how a moment on a circle is written — so .Cyclical("start", Period.HourOfDay) follows it too. Nothing of it reaches the file: each column is written as the kind that lands it there, so a pipeline that says the range and one that names the kinds are the same text. A range said twice, or said below a step it would have governed, is refused where the second is written.

Two choices are made here rather than discovered later:

• Outside the learned range, while the model is running: OutOfRange.Pass lets the value through, OutOfRange.Clip holds it at the edge, OutOfRange.Refuse says the data has moved. A pair the scale cannot honour is refused where it is written: Standard, Robust and Power land in no range and only pass, and Quantile holds at the edge or refuses, but cannot let a value through.
• The answer. Normalise what you predict and the way back is part of the saved pipeline: prepared.BackToOriginal(prediction) puts a prediction back in the units it was read in, walking every step that touched the answer backwards — and on through the column an answer was made from. The run checks that the way back really leads back, on every row, before handing anything over.

.ClipOutliers(column, Bounds.Iqr) holds the extremes to bounds learned from the training rows — by quantile, by spread or by the middle half — and what happens beyond them is your choice: Outlier.Clip, Outlier.Blank or Outlier.Refuse. Several columns at once take the same shape as the scales: .ClipOutliers(bounds => bounds.Iqr("fare", "age").Quantile(0.01, "volume")), with Outlier left on the verb that takes one column, since it is a decision about that column. A quantile bound at 0 is the smallest and largest value the training rows hold, so with Outlier.Refuse a later row beyond anything the fit saw stops the run and says how far out it was.

.NormaliseRow(Norm.L2, columns) is a different animal: it works across a row rather than down a column and learns nothing, which is what you want when the direction of a row matters and its size does not. A column is named once in it.

Evidence

using DeepSharp.Pipelines;

var looked = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "pclass").Optional("age", ColumnKind.Number).Number("fare"))
    .Profile()
    .Correlation(["survived", "pclass", "age", "fare"])
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Build()
    .Run();

var profile = (DataProfile)looked.Evidence[2];

Evidence is declared with the pipeline, before the numbers exist, so every run produces the same proof and a report cannot shrink to whatever happened to look good. Both kinds are measured on the rows the split trains on — the split below them as much as one above — because a profile over every row lets the rows a model is measured on shape what it is shown.

A profile gives, per column, the rows, the gaps, the values that are not numbers and the distinct values, and for numbers the smallest, the largest, the mean and the median. For each thing it finds it says the columns it is about and how it is answered: a gap among numbers with fill.missing, among words with the encoder, which makes a gap no category and marks it, and among true and false or moments with drop.gaps; a value that is not a number with fill.nan; words with an encoder. It names what should not be there, too, each answered by leaving the column out: a column that never changes; one that goes with the answer value for value, handing a model the answer — alive beside survived, with every value and how many training rows hold it; one that says again what a column before it says — class beside pclass; and one whose values are each a row's own, words no two rows share or a running number. A number far from every other, held by more than one row and written as files write that nothing is known — the twelve training fares of 0 — is answered by saying so in the schema, with missing. An extreme raises nothing: clipping or refusing one is a step you declare. It also counts the rows that are there more than once. The pipelines sample in the repository shows it on the passenger list: what each column's cells propose, then a profile of every column the file holds, each finding with its answer.

A correlation says which training rows it may be drawn from — those with a number in every column named — how many that is, and the rule that left the others out, because a correlation over the rows that happened to be complete is a different number from one over all of them. The pipeline works the correlation out, once: correlation.Correlate() gives a CorrelationMatrix of Pearson's coefficients and of Spearman's, which are Pearson's of the ranks, equal values sharing the average of the places they span. A pair with no coefficient — a column that never changes — is not a number, never nought. .Correlation(columns, Shown.Numbers, Coefficient.Spearman) says which one is shown, and DeepSharp.Charts draws it as a heatmap on the whole of the scale from minus one to one, correlation.Heatmap() or matrix.Heatmap(Coefficient.Spearman), as the Notebook draws or tabulates the same numbers.

A profile can be told to look for columns that keep one another's order: .Profile(rankAbove: 0.9) flags two columns of numbers whose Spearman coefficient, over the rows that hold a number in both, is above that in size, and says how many rows that was. The threshold has no default, because the figure at which two columns are too alike is put anywhere from 0.4 to 0.85 in published practice. An alert is one for each column, with its strongest earlier partner, and is answered by leaving that column out.

What they produce is in PreparedData.Evidence, by the step's place, and never in the pipeline's file: evidence is output, not the pipeline. A replay produces none.

The data after any step. pipeline.ViewAt(steps) gives the table as it stands after any number of steps, with where every row stands — in the part the split puts it in, dropped before it gets there, or undivided when nothing divides it. It is what the grid under a notebook block shows. MeasuredValues(column) and MeasuredCategories(column) say what its measured rows hold, by the rules a fit and an encoder read them by.

The handover

Batch(part) is where the pipeline stops: rows of numbers, their column names in a fixed order, and the answers handed over separately — Answers, as many numbers a row as the output names, in its order, with AnswerNames beside them; an output of one answer hands it over as Labels too. A network built here, a trainer from ML.NET or another .NET library, and something you wrote all take the same handover, which is the only reason two of them can honestly be compared. A column still holding words, a gap, a value that is not a finite number, and a row whose answers its kind of output refuses are refused there rather than passed on.

A network takes every feature on one scale, and says so: Batch(part, Needs.OneScale), or Served(rows, Needs.OneScale), refuses every feature that is not declared to land between minus one and one — scaled with MidRange, MinMax or MaxAbs, a moment on a circle in any form, a category encoded one-hot, true or false, a row divided by its size, a mark where a gap was. It reads where each feature lands from the declaration rather than the rows, so a pipeline loaded from its file is held to the same answer as the one you fitted. Declared to land there is what it is: the training rows land between minus one and one, and a row that arrives later and lies beyond what they held passes outside, unless its scale is declared to clip it or refuse it — on the Apple prices, four closes and one volume of the 152 days after the training rows do.

A pipeline read from its file holds no rows — its file keeps what the steps learned, not the rows — so it has no part to hand over and nothing for the report to measure, and asked for either it says so, naming the run over the rows that has them, new Pipeline(declaration, rows, folder).Run(). A row served through it is answered as through any fit.

Replay(rows) runs the same declaration over rows nobody had seen, with the numbers the training rows produced and nothing fitted again. It walks the steps exactly as the run did — it puts the rows in order and drops the warm-up rows too. Served(rows) hands the result over the way Batch does, without an answer — a served row is the question — and with which of the handed-in rows each served row is, and its key, so a prediction finds its way back. A served row is keyed by what it carries, its answer too when it is handed in with one, and served alike either way; one that lacks a column the pipeline reads is refused, with the columns the rows were handed in with.

A batch also says which part it was handed over from, Part, and which row each is, by its key, Keys, so what a model predicts for it is measured against the rows it was made for.

The way back, in your own units. BackToOriginal(predictions, part) puts the predictions for a part back, and BackToOriginal(predictions, served, rows) the predictions for served rows: each row is found again by where it was handed in and checked by its key, so rows handed in again in another order are refused rather than answered with another row's numbers. A way back that needs the row a number belongs to — a share of a flock, a return on a price — reads that row as it was read.

The pipeline as a file

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .FillMissing("age", With.Median)
    .Build()
    .Run();

var text = prepared.ToJson();                                               // both halves
var again = PreparedData.FromJson(text, StepCatalog.BuiltIn());             // anywhere, no data needed
var declared = PipelineDeclaration.FromJson(text, StepCatalog.BuiltIn());   // the person's half alone

A built pipeline is saved as one file with two halves, because they have two different authors: the declaration, which is what you wrote, and the fitted half, which is what the fit learned.

{
  "version": 8,
  "declaration": [
    { "step": "read.csv", "path": "titanic.csv" },
    { "step": "declare", "remainder": "drop", "columns": [
        { "name": "survived", "kind": "integer", "optional": false },
        { "name": "age", "kind": "number", "optional": true } ] },
    { "step": "split.stratified", "column": "survived", "train": 0.7, "validation": 0.15, "test": 0.15, "predict": 0, "seed": 20260923 },
    { "step": "fill.missing", "column": "age", "with": "median" }
  ],
  "fitted": [
    { "step": "split.stratified", "prefix": "…",
      "learned": { "rows.predict": 0, "rows.test": 135, "rows.train": 623, "rows.validation": 133, "digest": "…" } },
    { "step": "fill.missing", "prefix": "…", "learned": { "gaps": 133, "value": 29 } }
  ]
}

Keeping the halves apart is what lets you re-fit the same declaration on fresh data, compare two runs by their declarations alone, and run a trained model in a service that never knew the builder existed. Every fitted entry carries a key made from its own step and every step above it, so a fit is never used under steps that changed after it was learned.

Not everything in that half is a value the replay reads. A fit learns a value — a median, a bound, the categories it found — and a replay reads it. It decides something from what the training rows held, such as whether a column with that many gaps is filled at all, and a replay reads that too. And it sees: gaps above is a count of what it was looking at, written down for a person, and nothing replays it. The file marks none of the three, since it holds one object of names and values; the fit says them apart where it writes them, and the library is held to one rule because of it — nothing a replay reads was only counted. The split writes an entry of its own: how many rows went to each part, and a digest of the rows it divided. A run made for a learner that does without some steps writes a third block, skipped: each step it left out, by its verb and its prefix, so a replay leaves out the same steps. A run that leaves nothing out writes no such block.

A pipeline can also stand inside a larger file, as the value of one of its keys — a trained network and the pipeline it was trained behind are one file — and PreparedData.FromJson(text, StepCatalog.BuiltIn(), "pipeline") reads it there, exactly as its own file is read, every fault at its line in the larger file.

A file is read with a catalog — the list of verbs the reader knows. StepCatalog.BuiltIn() knows this package's own; add .WithIndicators() for a file that holds indicators, and .WithParquet(), .WithExcel() or .WithJson() for one whose rows those readers read, and .WithNetworks() for one that names the network its rows train. A step nobody registered is refused rather than skipped, and so is a key a step does not take. Rows landed from a live source need nothing added: what the file names of them is an ordinary read.csv, and StepCatalog.BuiltIn() reads it.

Everything wrong with a file is said at once, each fault at its line and column:

btceur.pipeline.json(9,5): Step 4: 'feature.indicator' is a step from DeepSharp.Pipelines.Indicators, which is not registered here. Reference the package and register its steps with the catalog that reads this file.
btceur.pipeline.json(10,5): Step 5: The step 'split.byTime' cannot be read: The shares add up to 1.3 and a split has to use every row.
btceur.pipeline.json(11,5): Step 6: 'normalize' is not a step anything here knows. The nearest one it knows is 'normalise'.

"I do not know this step" and "I know it, but it is not installed" are two different problems for the person reading, so they are two different messages.

The file names its version, and every verb the version from which it means what it says now. A file from 0.2 names none and is read as the first version: its declaration loads, except for the verbs whose meaning has changed since — the three splits and drop.warmup — which are refused by name rather than run the new way. A fitted half from 0.2 is refused: fit again. A file newer than the library is refused whole. The fourth version came with skipped: a run that leaves nothing out writes nothing else new, so its fit is the fit the third wrote, and a file of an earlier version that names skipped is refused at that key. The fifth version came when a declaration began to name its learner: learn.network means what it says from it, and a file of an earlier version that names it is refused at that step. The sixth came with the verb that settles a gap above the split: settle.gaps means what it says from the sixth, and a file of an earlier version that names it is refused at that step. The seventh came with the second learner a declaration can name, learn.ml, and the same holds for it. The eighth came with read.join, target.numbers and feature.cycle, which hold in the same way, and with the optional keys on verbs that existed — ordered and remainder, testSeed, id, coefficient, rankAbove, schedule and clip — each of which means what the file meant by not having it and is not written when it holds that meaning; 0.8.0 refuses a file of the eighth whole. The ninth came with the optional settings of the parts a declared network is made of — a dense layer's bias, weights, biasStart and rules, a normalisation's scale and shift, an optimizer's options, and the clipValue, stepEvery and average of the step — each of which means what the part meant by not having it and is not written when it holds that meaning; 0.9.0 refuses a file of the ninth whole. A file 0.5.0 wrote is read as it always was, and a network's file is held to the pipeline it carries by the digest of that pipeline's text, which leaves the version out, so a model trained behind a 0.4.0 or 0.5.0 pipeline loads and serves unchanged.

The schema and the template come from the same description the chain is built from. The JSON Schema of the file — pipeline.schema.json — the template a new step starts from, the verb reference and the notebook's form are all generated from each step's own list of parameters, so a file cannot be valid in a way the code is not.

Changing the columns, and saving what was decided

What can be done to a column is said once, for every door that changes the columns — the notebook's grid and its list of the source's columns, a take-over, and code:

using DeepSharp.Pipelines;

var declaration = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "pclass").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Normalise("fare")
    .Declaration;

var left = new PipelineDeclaration(declaration.Excluding("pclass"));                         // excluded, its kind kept
var back = left.Including("pclass", ColumnKind.Text, CsvRowSource.HeaderOf("titanic.csv"));  // back as it was
var rows = declaration.ChoicesFor(["survived", "pclass", "fare"]).Rows;                      // how each stands

Each operation hands back the steps the change makes and judges nothing: the rules every declaration keeps do that, so what the rules refuse and what is offered cannot disagree. Leaving out a column no step reads excludes it in the schema; leaving out any other drops it below the last step that reads it and the step that makes it. Including also takes several columns in at once, with one kind: one that breaks a rule stops the rest, and the steps it made are handed back for the rules to judge. WithKind changes a kind, and a category remembers the kind it came from; WithFormat says how a timestamp's moments are written, and WithMissing which value stands for a gap. ChoicesFor says how each column stands — taking part, excluded, dropped, made by a step and which one, kept with the rest, or not declared, and whether the source may lack it — what can be done to it without breaking a rule, and what it is to the output. An output is made under its verb by catalog.Make(verb, stated, carrying) and placed by WithOutput, in the place of the one standing and a return directly after the split, or taken away by WithoutOutput.

What a pipeline decided about its columns can be saved on its own, as a preset: the schema, the columns dropped, the output, and the source's columns as they were last shown — no fit and no rows. PipelinePreset.Of(declaration, header) takes it, ToJson() writes it, and PipelinePreset.FromJson(text, catalog) reads it back through the same door as a pipeline file. preset.TakeOver(pipeline, header) makes the steps those decisions make of another pipeline and lists, before anything is applied, every column whose decision changes — its kind, the kind a category was, whether the source may lack it, how its moments are written, which value stands for a gap — the output before and after, the schema's order and what it does with the columns it does not name when those change, the source's columns the preset never showed, and every saved drop it cannot make because its column no longer reaches the end, with how that column stands; steps that would break a rule are refused with every fault. CsvRowSource.HeaderOf(path) reads a file's first line alone.

A chain takes saved decisions over in the two places it is written in: the schema where it declares its columns, and the drops and the output where it names its answer, once the steps that read and make those columns are there. Each says what it decides before anything runs:

using DeepSharp.Pipelines;

var saved = PipelinePreset.FromJson(File.ReadAllText("titanic.columns.json"), StepCatalog.BuiltIn());

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(saved, out var columns)       // the saved schema, and every column it decides
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .FillMissing("age", With.Median)
    .Output(saved, out var rest)           // the saved drops and output, and what they change
    .Build()
    .Run();

Starting from a course

Every verb above can be written from memory, and a course says in which order: every step of a prepared pipeline, each present and waiting for what only you know. PipelineCourse.Table is the course for rows that do not depend on one another and PipelineCourse.SeriesInTime the one for rows that follow each other in time. Say fills a step in, Waiting names what still waits, and Pdd.From(course, catalog) starts a pipeline from a course filled in all the way, refusing one that is not with every fault at its step. A course is also a file of its own — ToJson() writes it and PipelineCourse.FromJson(text, catalog) reads it back — through the same door as the pipeline's file and under the same version. Starting from a course has both courses step by step, why each step stands where it does, and the code.

What the run must prove about a model

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close", "AAPL.Volume"))
    .OrderBy("Date")
    .SplitByTime("Date", train: 0.70, validation: 0.15, gap: 5)
    .Ahead("AAPL.Close", 5, AheadAs.Return)
    .Drop("Date")
    .Report(report => report
        .Measure(Metric.Rmse, Metric.Mae, Metric.R2)
        .On(Part.Train, Part.Validation, Part.Test)
        .As(Shown.Numbers, Shown.Drawn))
    .Build()
    .Run();

The profile and the correlation are evidence about the data; the report is evidence about a model, named before the numbers exist, on the three parts side by side — a low error on training beside a high one on test is overfitting, seen rather than deduced. In a file it is evidence.report, new in its third version, with metrics, parts and shown. There is one at most, below the output whose answers it measures, and a pipeline that names no answer or never divides its rows has nothing to measure. It measures the training, validation and test rows, and no others. A measure that counts classes — accuracy, precision, recall, the confusion matrix — is refused where it is written against shares of a whole or a return, which are amounts; against a column of answers it is checked when it is measured, and a row that is neither nought nor one is refused, named as it was read. The measures that compare shares — Metric.Emd, Metric.Kl and Metric.Rps, new in the eighth version — are refused the other way round: against any output that is not a distribution, and Emd and Rps, which follow an order, against one that does not say its columns are in one.

Metric.Emd	How many bands the predicted shares have to move to be the answer's, averaged over the rows: scipy's Wasserstein distance with the places of the bands as positions. A band's width is not in the declaration, so the unit is a band.
Metric.Rps	The ranked probability score: the squared distance between the cumulative shares, added up over the thresholds.
Metric.Kl	The Kullback–Leibler divergence from the prediction to the answer, in nats, with no epsilon added; infinite where the prediction holds nothing the answer holds.

Each row is divided by its own total first, so a distribution that comes back as counts is compared as shares, and each is measured beside the same measure of predicting the training rows' average shares.

The run acts on nothing for it: the measures are taken once a model has predicted. prices.Measure(predictions) takes what a model predicted for each part it names — for the rows Batch(part) handed over, in their order — and gives the measures, each in the answer's own units, since the predictions and the answers both come back through the way back: in normalised units every error is small and every model looks excellent. Here a predicted return is measured as the price it comes back as. Beside each measure stands what predicting, for every row, the average of the training rows' answers would score, so no number stands alone; and the measures of amounts and of classes are scikit-learn's, the three that compare shares scipy's and xskillscore's. Predictions for other rows, or for the right rows in another order, are refused: each would be measured against another row's answer, and nothing in the numbers would say so. The report reads only the keys and the answers of the rows it measures, never their features. A network trained behind the pipeline is measured by its report without being asked — see Networks — and what is measured is output, never written into the pipeline's file. DeepSharp.Charts draws it: the measures as bars beside the average, each confusion matrix as a heatmap of counts, what was predicted against what was there, and what was left over; and measures.Report() renders the whole report as HTML, as it says the measures are shown.

What was predicted can cross as text. measures.PredictionsToJson() writes the fit the predictions were made behind, and for each part the key of each row, what was predicted for it and what the model said it learned nothing about; and prepared.MeasureAgain(text) measures that text again, by the report, on another run of the same pipeline. It is held to the fit as a network's file is held to its pipeline: predictions made behind other steps or other learned numbers — a feature or the answer scaled another way, another split — are refused, as are those of another output, named by its answers, and of other rows, known by their keys. A notebook's report block draws a model a C# cell trained this way — Notebook.

Declaring the learner

The pipeline stops at the handover, and what learns from the rows used to be written beside it: the steps were declared, a network was built and fitted by hand, and two runs of one declaration could train different models while the file said nothing about which. A declaration can name the learner its rows are prepared for instead, below the output it learns to answer, and the file then says what was trained behind its steps:

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")      // a network takes them between -1 and 1, which is the default
    .Report(report => report.Measure(Metric.Accuracy).On(Part.Validation, Part.Test).As(Shown.Numbers))
    .WithTensorflow(network => network                  // the network these rows train
        .Dense(16).Relu().Dense(1)
        .Adam(0.01)
        .BinaryCrossEntropy()
        .Run(seed: 20260929, epochs: 100)
        .StoppingAfter(patience: 10))
    .Build();

var trained = pipeline.Train();                         // the steps, and then the network they name

A trainer from ML.NET is named here too, with .WithML(trainer => trainer.FastTree()), and pipeline.TrainWithML() trains it — one learner a declaration, so a tree and a network are two declarations differing in that line, measured by one report on the same rows. See ML.NET learner.

The chain says it in the words TensorFlow and Keras use with .WithTensorflow(…), or in the words PyTorch uses with .WithTorch(…). Both write the same step and lower onto the same model, and where a caller leaves a number unsaid each writes the number its own library leaves there, so the file says which run was asked for rather than which library asked for it. Networks has the words, the numbers each leaves and what Train() hands back.

In the file the step is learn.network, new in the fifth version. The layers, the optimizer, the loss and the stopping are each a name and the settings that name takes, and the engine, the seed, the epochs and the batch size of the run stand beside them:

{ "step": "learn.network",
  "layers": [ { "kind": "dense", "units": 16 }, { "kind": "relu" }, { "kind": "dense", "units": 1 } ],
  "optimizer": { "kind": "adam", "rate": 0.01, "firstMoment": 0.9, "secondMoment": 0.999, "epsilon": 1e-07 },
  "loss": { "kind": "binaryCrossEntropy" },
  "stopping": { "kind": "patience", "patience": 10, "least": 0, "best": true },
  "engine": "light", "seed": 20260929, "epochs": 100, "batch": 32 }

Every setting a name takes is written down, and a file that leaves one out is refused where it is read: a declaration is replayed from what it says, and a number nobody wrote would make two runs of one file differ as soon as a default moved. The layers are a list, from the one the rows reach first; the optimizer, the loss and the stopping are one part each, written as itself; and engine is only a name, light unless said, which engine it stands for being the application's to say. Five keys stand beside them only when they are said. schedule, a part that says how the rate changes from epoch to epoch (constant, stepDecay, exponentialDecay, cosineDecay or linearWarmup), and clip, the most norm the gradients of a step may have together, are new in the eighth version; clipValue, the most any gradient value may reach, stepEvery, how many batches go into a step, and average, the share an average of the weights keeps, are new in the ninth. Left out they mean a constant rate, no clipping, a step after every batch and no average, and they are not written. The settings the ninth version added to the parts are optional in the same way: a dense layer's bias, weights, biasStart, l1, l2, maxNorm and learns, a batchNorm or layerNorm's scale, shift and learns, and an optimizer's weightDecay, dampening, nesterov, amsgrad and centered. Each left out means what the part always meant, and is not written when it holds that meaning — which is why a declaration that says none of them writes the file it always wrote. Networks has what each does and when to reach for it.

A declaration has one learner at most, and it stands below the output it learns to answer: a learner with no output, or above it, or a second one, is refused with the step it is at, as every rule on this page is. The step changes no column and runs nothing — what it names happens once the pipeline has run, in the package that brings the verb — so DeepSharp.Pipelines still references no learner, no engine and no network. It says one thing the pipeline itself uses, what the learner needs of the features it is handed, which for a network is Needs.OneScale, and PipelineDeclaration.Learner is where every part that needs it asks.

pipeline.Train(), which DeepSharp.Learners.Networks brings, runs the pipeline for the learner it names — leaving out the steps that learner does without, as RunFor does — trains the network, measures it as the report declares and hands back the TrainedNetwork that Networks describes; prepared.Train() does it for a run already in hand. pipeline.Train(engines, onEpoch, token) hands every epoch over as it ends and stops when the token is cancelled, throwing and returning nothing, and prepared.Train(step) trains the network a step declares on rows already prepared — which is how a search tries many networks on one preparation: see Searching for a network. Run() answers what it always answered, the prepared rows, and a pipeline that names no learner says so when it is asked to train, rather than guessing one.

The verb comes with that package, as read.parquet comes with its reader: a catalog reads a file that names it once it is taught, StepCatalog.BuiltIn().WithNetworks(), and one that never was says which package brings it rather than that the verb is misspelled. The notebook's own catalog knows it.

Choosing what learns

Everything up to here is the same whatever is going to learn from the result, so this is where the pipeline stops. What learns is your choice — a network built here, a trainer from an established .NET machine-learning library such as ML.NET, Microsoft's own, or one you wrote — and they all plug in at the same handover. The network is built: it takes Batch(part, Needs.OneScale), every feature declared to land between minus one and one, and Networks has it. A trainer from ML.NET stands on the same seam, and ML.NET learner has it; the seam itself is below.

That is what makes a comparison mean anything. Two learners measured on the same prepared data with the same declared measures can be put beside each other; two learners each fed by their own preparation cannot. The same data is not the same representation for everyone, though: a network wants everything numeric, expanded and on one scale, while a tree does not care about scale and is hurt by a wide one-hot column. So a learner says what it needs, and the steps it does not need are skipped by declaration and written down as skipped — never dropped quietly, because the artefact has to keep saying what each run actually saw.

Needs names four needs, one value each:

Needs.Numbers	Numbers only, each on the scale its steps declare: every step as declared, as a linear model takes it.
Needs.OneScale	Every feature declared to land between minus one and one, as a network takes them.
Needs.NoScale	Numbers of any size, as a tree takes them: a step that only scales a feature is left out.
Needs.Categories	Each category as its place in the list the training rows held, with numbers of any size, as a boosted tree that splits on categories takes them.

using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")      // a network takes them between -1 and 1, which is the default
    .Build();

var forATree = pipeline.RunFor(Needs.Categories);
var left = forATree.Skipped;                              // the places of encode.categories and the four normalise steps
var train = forATree.Batch(Part.Train, Needs.Categories); // nine numbers a passenger, with their categories as places

pipeline.RunFor(needs) runs the declaration for that learner, and Run() takes every step, as it always did. A step that only scales a feature is left out for a learner indifferent to scale, and the encoder hands each category over as itself to a learner that takes categories: the place of each in the list the training rows held, the place after the last for one they never held, and minus one for a gap — the same list it learns for every learner. Batch.Categories and ServedBatch.Categories name each feature handed over as a place, with the categories the training rows held. A step an answer's way back runs through, a step whose column a step below reads, and a scale that refuses what it was not fitted on are always taken, whatever the learner needs: leaving them out would change more than what that learner does without. What is left out is decided from the declaration and the need alone, never from the rows; Skipped lists it, the file writes it under skipped, and a replay leaves out the same.

On the Titanic pipeline above, a passenger is handed over as nine numbers for Categories — age 29, fare 8.4583, class as place 2 of 3, sex as place 1 of 2 — and as fourteen for the network. Both runs share the split, the filled age and the categories learned, so the report measures both learners on the same 623, 133 and 135 rows, and predictions handed back as text from either are measured again on a run of every step. A learner is handed only a run it can take: Batch and Served refuse a run that left out a step the learner needs, naming each, and a network is refused a run made for NoScale or Categories.

A step of your own that some learners do without says so by IMeetsANeed — NeededBy(needs), whether a learner of that need takes it as declared, and Instead, what a run for the others takes in its place, or nothing. Only a step that learns can: a source, a split, a drop, an order or an added column decides what the data says for every learner alike. How a step of your own is written, registered and read back from a file is on Writing a step of your own.