Splitting the rows - xkqg/DeepSharp GitHub Wiki

Step 5 — Splitting the rows

In short

The split is the line the library is built around. Above it stands everything that is the same for every row, whoever reads it; below it, everything learned from data. The one rule to remember: only the training rows may teach what is learned, and no row is taken out below the split.

You want to…	Write
keep the shares of a rare answer the same in every part	`.SplitStratified("survived", train: 0.70, validation: 0.15)`
learn from the earliest rows and measure on the latest	`.SplitByTime("dteday", train: 0.70, validation: 0.15)`
keep a gap of moments between the parts of a series in time	`.SplitByTime("Date", train: 0.70, validation: 0.15, gap: 5)`
divide rows that do not depend on one another at random	`.SplitAtRandom(0.70, 0.15)`
make every slice of a small table validation once	`.SplitAtRandom(0.80, new Fold(5, index))`
move the cut of a series forward and fit again	`.SplitByTime("Date", 0.80, new Origin(3, index))`
mix a file written in an order before it is divided	`.Shuffle(seed)` above the split
put a series in time in its order	`.OrderBy("Date")` above the split
set rows aside to calibrate an interval on	`.Calibrate(0.2)` before the split
ask how many rows a part holds	`prepared.CountIn(Part.Train)`

The rest of this page explains each one.

What the step looks like

The passenger list is divided with the share of survivors kept the same in every part, and the line in the code marks where learning begins.

using DeepSharp.Pipelines;

var passengers = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)   // test is what is left
    // ---- nothing above this line learns from the rows ----
    .Target("survived")                                           // the answer, named right below the line
    .FillMissing(fill => fill.Median("age"))
    .Normalise("age", "fare")
    .Build()
    .Run();

Console.WriteLine($"{passengers.CountIn(Part.Train)} train, {passengers.CountIn(Part.Validation)} validation, {passengers.CountIn(Part.Test)} test");

Seventy, fifteen, and what is left

Seventy to learn from, fifteen to choose with, and what is measured on is never written down — it is what is left.

There is no test share to name: the pipeline's file writes it down, as what is left, so nothing can ask for more rows than there are.

A share is an aim; the parts hold what was achieved

Rows that belong together are never pulled apart:

the same row twice goes to one part;
a split by time keeps every row of one moment together, so a moment with many rows — a day with a row for each flock — lands whole in one part;
a gap takes whole moments as well.

The parts are then as near the shares as whole moments allow, not exactly them.

What each part really holds is written down:

ask the run with CountIn(Part.Train);
or read rows.train, rows.validation, rows.test and rows.gap in the fitted file;
for a split by time, moments.train, moments.validation and moments.test give the first and the last moment of each part.

Which split, and why it matters more than it looks

Split	What it does	When
stratified	keeps the shares of an answer the same in every part	one answer is rare
by time	gives the earliest rows to learn from and the latest to be measured on, with a gap of days between the parts	a series in time
at random	deals the rows by a seed	rows that do not depend on one another

The gap of a split by time is there so that a row just before the boundary cannot leak into the part just after it through a feature that looks ahead.

Using a random split on a series in time is the most common way to get a wonderful validation number and a worthless model.

Folds: every slice of a small table is validation once

A small table is measured by the luck of one division. A Fold(5, index) cuts the rows that are neither test nor predict into five slices of about as many rows, and makes slice index the validation rows and the other four the training rows:

using DeepSharp.Pipelines;

const int Folds = 5;

Pipeline FoldOf(int index) => Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitAtRandom(0.80, new Fold(Folds, index), seed: 100, testSeed: 7)   // no validation share: the fold is validation
    .Target("survived")
    .FillMissing("age", With.Median)
    .Build();

var prepared = Enumerable.Range(0, Folds).Select(index => FoldOf(index).Run()).ToArray();

The test rows do not move. They are dealt by testSeed alone, so every fold is measured on the same rows.
A repeated record stays in one slice, as it stays in one part.
Every fold is a pipeline of its own, with a fit of its own: what a fill or a scaling learns, it learns from that fold's training rows. A network trained behind one fold is refused beside another.
A validation share beside folds is no use to them: the fold says what validation is, so the share only adds to the rows that are cut. Leave it out.
A file says it with two keys, folds and fold, and a file that says only one of them is refused.
SplitStratified takes a fold as well, and each group of the column is cut into slices.

A series in time is not dealt again; the cut is moved. new Origin(3, index) cuts the rows that learn, in the order of their time, into four blocks. Origin index trains on the first index + 1 of them and validates on the next, so what a model learned from always came before what it is chosen on; the blocks after validation are fitted on by nothing, and the test rows are one block at the end. A gap keeps the last moments of training and of validation apart, and a moment is never divided. The keys are origins and origin.

What the folds measured is laid beside itself in Measuring and drawing, and a study takes the folds as its deals in Searching for a network.

Rows set aside to calibrate an interval on

Calibrate(0.2) before a split sets a fifth of the rows aside as a part of their own. Nothing is fitted on them and nothing is chosen by them, so the errors a model makes there are the errors it will make on rows it has not seen — which validation cannot give, since a model is chosen by it, nor test, which is reported on once everything else is settled.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Number("fare", "age").Integer("sibsp"))
    .Calibrate(0.20)                                          // a fifth of the rows, a part of their own
    .SplitAtRandom(0.50, 0.10, seed: 3, testSeed: 7)          // the calibration rows are dealt by the test seed
    .Target("fare")
    .Drop("age")
    .Build()
    .Run();

Console.WriteLine(prepared.CountIn(Part.Calibration));

A split by chance deals them by the test seed, so they are the same rows on every deal, like the test rows.
In a series they come just before the test rows, the most recent of the rows that are not measured on.
A fit writes rows.calibration, and a split writes calibration, only when rows are set aside: a pipeline that sets none aside is the same text it always was.
Measuring and drawing has the interval worked out from them.

What the split writes down

How many rows went to each part, and a digest of the rows it divided that does not depend on the order they arrived in. So what a fit saw travels with what it learned, and a replay can be held to it.

A file's own order is not a mixture

Rows written every survivor first, or every month in turn, give parts that are not alike. .Shuffle(seed) above the split puts them in an order drawn from that seed so each part holds the same mixture; without it they keep the order they were read in.

A series in time is the exception that proves it: there you write .OrderBy, and the split by time wants exactly that order.

Rows are divided here and never taken out below

Dropping a row below the split would quietly change how many each part holds, which is the one thing the split promised. That is why every verb that removes rows stands above it.

Next: Naming the answer · All three splits and the predict share: Pipeline