Step 2 — Declaring the columns

In short

The schema is the one place that says what takes part. A column you do not name does not come along, and a column you do name is read as the kind you gave it or the run stops. The one rule to remember: only a column declared Optional may have an empty cell.

You want to…	Write
declare whole numbers	schema.Integer("survived", "sibsp", "parch")
declare numbers that may not be absent	schema.Number("fare")
declare words from a fixed set	schema.Category("pclass", "sex")
declare a column whose cells may be empty	schema.Optional("age", ColumnKind.Number)
declare a date and time	schema.Timestamp("dteday")
mark the id a team keeps beside its rows	schema.Id("flock", ColumnKind.Integer)
keep, or refuse, the columns the schema does not name	.Declare(schema => …, Remainder.Keep) or Remainder.Refuse

The rest of this page explains each one.

What the step looks like

The schema below names seven columns of the passenger list, and only those seven come along. This is deliberate: a file that grows a column next month must not quietly grow a feature.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema
        .Integer("survived", "sibsp", "parch")       // whole numbers
        .Category("pclass", "sex")                   // words from a fixed set
        .Optional("age", ColumnKind.Number)          // a number, and it may be absent
        .Number("fare"))                             // a number, and it may not
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Build()
    .Run();

Console.WriteLine($"{prepared.Table.Columns.Count} columns of {prepared.Table.RowCount} rows");

Optional is the word that matters

It says a cell may be empty — and only an optional column may be.

A gap in a column you declared Number stops the run at the row it found, with the row's number. A number that is sometimes absent is a different thing from a number, and a model trained on one and served the other is the bug this whole library is shaped around.

The columns you did not name

The columns the file holds and the schema does not are dropped by default. Saying so differently — keeping them, or refusing a file that brings an unknown column — is one word on the schema, described on Pipeline.

A column can be the id

schema.Id("flock", ColumnKind.Integer) marks the number a team keeps beside its rows. It is carried with every row and is nothing else — not a feature, not encoded.

It comes back as Ids beside the rows handed over and beside a network's predictions, so an answer finds its row by what the row says.

What the row looks like after this step

The empty age is now an absence the pipeline expects: declared optional, still empty. Every later step knows the difference between "nobody wrote it down" and "it happened to be nought".

Next: Settling the gaps · Every kind and what refuses what: Pipeline