Step 8 — Measuring and drawing

In short

The measures are declared with the pipeline, before a single number exists, and the charts are drawn from what the run already kept. A measure chosen after seeing the result is a measure chosen because of the result; declared first, every run is measured the same way whether it turned out well or not. The one rule to remember: choose the measures before the run, and never tune against the test rows' numbers.

You want to…	Write
declare a report before anything learns	.Report(report => report.Measure(…).On(…).As(…))
measure a class	.Measure(Metric.Accuracy, Metric.Precision, Metric.Recall, Metric.ConfusionMatrix)
measure how a model ranks its rows and how its chances hold	Metric.F1, Metric.Auc, Metric.Brier
measure an amount	Metric.Rmse, Metric.Mae, Metric.R2, Metric.Mape
measure shares in an order	Metric.Emd, Metric.Rps, Metric.Kl
say which parts are measured and how they are shown	.On(Part.Train, Part.Validation, Part.Test).As(Shown.Numbers, Shown.Drawn)
write the report as a page	trained.Measures!.Report().ToHtml()
draw the loss curve	trained.History!.LossCurve()
see how alike the parts are	.Likeness(warnAbove) below the split
keep the training rows a served model is watched against	.Watch(warnAbove) below the split
measure a window of served rows against them	prepared.Watched(window)
draw the windows one beside another	windows.Distances()
show a correlation as numbers or a picture	.Correlation(columns, Shown.Numbers, Coefficient.Spearman)

The rest of this page explains each one.

What the step looks like

using DeepSharp.Charts;
using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Category("sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing(fill => fill.Median("age"))
    .EncodeCategories()
    .Normalise("age", "fare")
    .Report(report => report                                  // declared before anything learns
        .Measure(Metric.Accuracy, Metric.Precision, Metric.Recall, Metric.ConfusionMatrix)
        .On(Part.Train, Part.Validation, Part.Test)
        .As(Shown.Numbers, Shown.Drawn))
    .WithTensorflow(network => network
        .Dense(16).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 20260929, epochs: 100))
    .Build();

var trained = pipeline.Train();

File.WriteAllText("titanic-report.html", trained.Measures!.Report().ToHtml());
File.WriteAllText("titanic-loss.svg", trained.History!.LossCurve());

Every measure comes with the average's beside it

A model that beats the average by nothing is a model that learned nothing, and that comparison is worth more than any single number.

The test rows' measures stand last, because they are the ones you are not allowed to tune against.

The report as a page

Measures!.Report() is the report as the declaration said its measures are shown, and ToHtml() is that page — which is also what a notebook block shows when you ask it for the report.

The charts are drawn from what the run already kept

• the loss curve
• the learning rate
• the confusion matrix
• predicted against actual
• the residuals
• the measures as bars
• the reliability curve
• the windows a served model met, each column's distance from the training rows

All are SVG, with no viewer to start and nothing recomputed. That is DeepSharp.Charts, which draws with MatPlotLibNet and is a package of its own: a project that serves a model and draws nothing never carries it.

Classes are measured by more than whether they are right

Measure	What it is
Metric.F1	the harmonic mean of precision and recall, label by label and averaged
Metric.Auc	the area under the curve a threshold sweeps out — the chance a row that holds a class is given more of it than a row that does not
Metric.Brier	the Brier score: the squared distance between each chance said and the class held

Metric.Auc is taken of the chances themselves before any threshold, so a model is measured on how it ranks its rows. The Brier score answers whether a model that says seventy per cent is right seven times in ten.

Drawn, the Brier score stands beside the reliability curve: the rows sorted by the chance said and dealt into ten bins of as many rows each, the chance said in a bin against how often its rows held the class.

All three are scikit-learn's, held to it to twelve places. What they refuse:

• A part in which every row holds a class, or none does, has no area under the curve and is refused; scikit-learn says the area is not defined there and gives no number.
• Labels of which a row holds any number are refused for the Brier score: scikit-learn takes each row for one class of many, warns that the chances do not add up to one, and gives a number that measures nothing these labels mean.

The expected calibration error is not built: binned, it misjudges most the models that are best calibrated, and the Brier score bins nothing.

An amount can be measured as a share of itself

Metric.Mape is the mean absolute percentage error, as a share: each error's size over its answer's.

A part that holds an answer of nought is refused, naming the row. Scikit-learn divides by the smallest number above nought instead, which makes that row's error as large as a number gets and the measure say nothing else.

How alike the parts are is evidence too

.Likeness(warnAbove) below the split sets out, column by column, how far apart training, validation and test are, each distance beside the rows it was worked out from:

• for a column of numbers, the two-sample Kolmogorov–Smirnov distance, ties taken whole as scipy takes them;
• for a column of words, the total variation — half the sum of how far each word's share differs.

A distance above warnAbove is flagged. The bound has no default, because how far apart is too far depends on how many rows a part holds.

It reads the features of every part and never the answer, and it warns and never refuses:

• a validation part unlike the training rows is a poorer guide to stop on;
• a test part unlike them explains a measurement that disappoints.

A notebook draws it as a table.

A served model is watched against the rows it learned from

The likeness of the parts is measured once, while every row is there. A model that ships meets rows nobody had then, and the pipeline it serves behind is read from its file, which keeps what the steps learned and not the rows they learned it from. .Watch(warnAbove) below the split is the step that keeps them: the training rows of every column it watches, each value once with how many rows hold it. It changes nothing a replay does.

prepared.Watched(window) then measures a window of served rows — a day of them, an hour, as you group them — against those rows, by the same two distances and the same rule as the likeness of the parts:

using DeepSharp.Charts;
using DeepSharp.Pipelines;

// The pipeline a model serves behind, read from its file: it declared .Watch(0.1) below its split.
var prepared = PreparedData.FromJson(File.ReadAllText("flocks.pipeline.json"), StepCatalog.BuiltIn());
var windows = new List<WindowLikeness>();

foreach (var day in Directory.GetFiles("served", "*.csv").Order(StringComparer.Ordinal))
{
    var window = prepared.Watched(new CsvRowSource(day));

    foreach (var column in window.Columns.Where(column => column.Above))
    {
        Console.WriteLine($"{window.From:yyyy-MM-dd}: {column.Column} is {column.Apart:0.00} away, over {column.WindowRows} rows");
    }

    windows.Add(window);
}

File.WriteAllText("watched.svg", windows.Distances());

• Each column says its distance, whether it is above the bound, and how many rows of the window and of the training rows it was worked out from; the window says the first and last moment it holds in the column its rows are ordered by.
• The window is replayed to where the watch stands, so a category the training rows never held is measured there, not refused.
• Nothing is learned from a window, and none is kept between calls: the series is yours, and windows.Distances() draws it — a line to each column, under the bound.
• There is no least window. A distance over a few rows is noise as much as anything, which the rows beside it say.

Left unnamed, the watch keeps every column but the id, the answer and the columns the rows are ordered or divided in time by, which every window holds otherwise than the training rows. The counts are exact, so a column of numbers with every digit keeps a number for every training row: name the columns to keep the file small. A model trained before its pipeline kept the rows is watched once the pipeline is fitted again with the watch and the model trained behind that fit.

An answer that is a distribution is measured along its order

For shares in an order — a flock weighed in bands:

Measure	What it says
Metric.Emd	how many bands the predicted shares have to move, on average over the rows, to be the answer's
Metric.Rps	the ranked probability score
Metric.Kl	the divergence from the prediction to the answer — infinite where the prediction holds nothing the answer holds, which the report writes and draws as such

The output says its columns are ordered, or a report that names Emd or Rps is refused where it is written.

R², RMSE and MAE still apply to a distribution, but they are averaged a column at a time, and a prediction one band off scores a low R² over hundreds of bands, which is why the others are there. Pipeline has the definitions.

A correlation is numbers before it is a picture

• correlation.Correlate() gives Pearson's coefficients and Spearman's between the columns, worked out once from the rows the correlation was kept over.
• .Correlation(columns, Shown.Numbers, Coefficient.Spearman) says which one a notebook shows.
• matrix.Heatmap(Coefficient.Spearman) draws it.

A column that never changes has no coefficient with any other, and its cells are blank in the picture and n/a in the table, never nought. A profile told .Profile(rankAbove: 0.9) flags two columns of numbers that keep one another's order.

Whether a change helped has a spread

The same network scores a few percent differently on two divisions of the same rows, as much as most changes move it, so one split cannot say. Searching for a network has the paired comparison that trains the network behind two pipelines on each of many deals and reports the differences with their spread.

A tree and a network are measured by one report

One learner a declaration, so the comparison is two declarations that differ in a single line — .WithML(trainer => trainer.FastTree()) where the other says .WithTensorflow(…).

Everything above that line is the same steps over the same rows in the same parts. The run made for a tree leaves out only the scalings a tree does without, and writes down which, so two learners fed the same rows are compared rather than guessed between.

The report then reads the rows' keys and their answers and nothing of the features, so the two are measured on the same rows, by the same metrics, in the answer's own units. See ML.NET learner.

Next: Naming the network · Every measure and every chart: Networks