Settling the gaps - xkqg/DeepSharp GitHub Wiki

Step 3 β€” Settling the gaps

In short

This step puts a value nobody learned into the gaps a feature will be worked out from, above the split. It exists because a feature made from a column with a gap is itself a gap, and a fill below the split cannot reach back into it. The one rule to remember: settle a column here for the features, or fill it below the split for the model β€” not both.

You want to… Write
put a nought in a column's gaps .SettleGaps(gaps => gaps.Zero("age"))
put a number you choose in them .SettleGaps(gaps => gaps.Constant(-1, "age"))
stop the run at a gap instead .SettleGaps(gaps => gaps.Refuse("age"))
take the rows with a gap out .DropGaps("age")
see which rows had a gap read the column age_was_missing the step writes
fill from the training rows, for the model .FillMissing(fill => fill.Median("age")) below the split β€” Filling and scaling

The rest of this page explains each one.

Why this step exists

This is the step most courses leave out, and the one the order exists for. A feature worked out from a column with a gap is itself a gap.

Filling that column later cannot reach back into the feature. A fill learns its value from the training rows and therefore stands below the split, long after the feature was worked out.

So the gaps a feature will be worked out from are settled here, above the split, with a value nobody learned.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp").Optional("age", ColumnKind.Number).Number("fare"))
    .SettleGaps(gaps => gaps                         // Zero, Constant(value), Refuse
        .Zero("age"))                                // a nought no row decided
    .AddFeature("older", "age", Arithmetic.Plus, "sibsp")
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("age", "older", "fare")
    .Build()
    .Run();

Console.WriteLine(prepared.Table.NumbersOf("age_was_missing").Count(value => value == 1));   // 177

Why this one may stand above the split

A nought, a number you choose, and a refusal are not values any row could have told you. No row of validation and no row of test taught them anything, so putting them in before the rows are divided leaks nothing.

The step writes nothing into the fitted half of the file, because there is nothing it learned. A replay does what the run did by doing it again.

A nought is not an absence, and the pipeline keeps the receipt

Beside every column it settles, the step writes <column>_was_missing, one per row. Settling destroys the difference between "absent" and "the value happened to be that" β€” permanently β€” and that column is the only record left that the difference existed.

A verb that marks the same column later adds to it and never overwrites it.

One answer or the other

A column settled here has no gap left, so a .FillMissing of the same column below the split can do nothing. The pipeline refuses that line where you wrote it rather than run a step that cannot act.

Settle it up here for the features, or fill it down there for the model; not both.

Dropping the rows instead

Dropping the rows is a verb of its own, .DropGaps, and it stands up here too, because rows are taken out before they are divided and never after.

Next: Working out the features Β· Every strategy and every refusal: Pipeline

⚠️ **GitHub.com Fallback** ⚠️