Step 7 — Filling and scaling

In short

Everything in this step learns a number from the training rows, replays it unchanged on validation, on test and on every row served a year later, and writes it into the file so it can be replayed at all. That is why it stands below the split. The one rule to remember: what is learned here is learned from the training rows alone, and a scaling whose bounds you already know is not learned and stands above the split instead.

You want to…	Write
fill gaps with the training rows' median	.FillMissing(fill => fill.Median("age"))
fill them another way	Mean, Zero, Constant(value, …), Previous or Refuse on the same line
turn every declared category column into numbers	.EncodeCategories()
scale columns to between minus one and one	.Normalise("age", "fare")
choose the kind of scaling per group of columns	.Normalise(scale => scale.Columns("age", "fare").MaxAbs("sibsp"))
land the features between nothing and one instead	.DefaultFeatures(Form.Unit) above the split
scale by bounds you know rather than learn	.ScaleGiven("fare", 0, 512) above the split
write a number as the bins it fills	.PiecewiseLinear("age", bins: 8)
give every row its key's history	.History(key: "flock", value: "weight", shrink: 5, "date", "round")

The rest of this page explains each one.

What the step looks like

This is the half of the course that a hand-written script almost always gets wrong.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    // ---- nothing above this line learns from the rows ----
    .Target("survived")                              // the answer, named below the line and before what learns
    .FillMissing(fill => fill.Median("age"))         // Mean, Median, Zero, Constant, Previous, Refuse
    .EncodeCategories()                              // every category column the schema named
    .Normalise(scale => scale                        // a kind a group of columns
        .Columns("age", "fare")                      // the default: between minus one and one
        .MaxAbs("sibsp"))
    .Build()
    .Run();

Console.WriteLine(prepared.Fitted[4].Number("value"));   // the median the training rows gave

One line, many columns, one kind each

Every per-column verb takes a line of its own — here below the split .FillMissing, .FillNaN, .ClipOutliers and .Normalise, and above it .Reshape, .Cyclical and .TimeParts.

What reaches the declaration is one step a column either way, so the file reads the same as if you had written the verb once per column.

Where features land is said once

A scaling that names no kind puts them between minus one and one, which is where a network takes its features.

.DefaultFeatures(Form.Unit) says nothing to one instead, once, above the split, and a column that names its own kind keeps it.

What Refuse is for

Several of these verbs offer a refusal as a strategy, and .FillNaN refuses by default.

A not-a-number is somebody's broken division, and carrying it into a model as though it were a measurement is the one thing a pipeline should never do quietly.

A scaling whose bounds you give stands above the split

Everything else on this page reads its numbers from the training rows, which is why it is here. .ScaleGiven("fare", 0, 512) is told them instead — a fare runs from nothing to the most anybody paid, an hour from nought to twenty-three — and that is knowledge about the column rather than about the rows.

So it:

• stands where the features are worked out, and a feature after it is already on the scale a model takes;
• writes nothing into the fitted half;
• has an exact way back without anything written down.

One line does many columns: .ScaleGiven(scale => scale.Between("fare", 0, 512).Between("age", 0, 100)).

A learner that does without some of this still gets it declared

A tree takes numbers of any size, so a run made for one leaves the scalings out — and writes down which it left out, in the file, so what that learner was fed is never a guess.

Declare them anyway: the same declaration then trains a network too, and the two are measured on the same rows.

What is never left out is the way back, since an answer has to come back in its own units whoever predicted it.

A number can be written down as the bins it fills

.PiecewiseLinear("age", bins: 8) learns the edges of eight bins from the quantiles of the training rows — edges that fall together counted once — and writes the column as one column per bin:

• one where the value lies beyond the bin;
• nought before it;
• the share of the bin in between.

It is Gorishniy et al.'s piecewise linear encoding, which lets a network see a number's shape the way a tree sees its splits.

• A value beyond the training rows' ends is let through by default, or held at them, or refused, as outOfRange says.
• A column that holds one value on its training rows has no edges, and is refused.

A key's history is a feature that learns

.History(key: "flock", value: "weight", shrink: 5, "date", "round") writes beside every row the mean of weight over the training rows of the same flock that come strictly before it. "Before" is on the order the date gives and, where a date is shared, the round. The mean is pulled towards the mean of every training row by five rows' weight.

• A flock the training rows never held gets that overall mean.
• The step keeps the training rows themselves, so a row of validation, of test, or one served on its own later is answered from them alone. A single new flock gets the history its key had, where a window over the rows handed in with it would have none.
• shrink has no default, since how far a key with few rows is trusted is yours to say.
• A row with a gap in an ordering column is refused, naming it.

The sharp edge, once more

A column settled above the split has no gap left to fill, so a fill of it here is refused at the line — see Settling the gaps.

Next: Measuring and drawing · Every strategy, every scale, every encoder: Pipeline