Searching for a network - xkqg/DeepSharp GitHub Wiki

Searching for a network

In short

A study tries several networks and keeps the one that did best: which learning rate, how many units, which activation. It does so by the rule this library is built on: every candidate is trained on the rows the pipeline prepared, chosen between by the validation rows, and never looked at on the test rows until one has been chosen. A Comparison asks the same question one level up: did a change to the pipeline help?

You want to… Write
say what may vary new SearchSpace([new LogRange("rate", 1e-3, 1e-1), new WholeRange("units", 4, 32), new Choices("activation", ["relu", "tanh"])])
turn a candidate into a network new Study(space, (candidate, network) => network.Dense(candidate.Whole("units"))…)
say how many trials and deals Trials = 12, Deals = 3
draw candidates from a seed Sampler = new RandomSampler(11)
draw a setting only under another's choice new NumberRange("momentum", 0, 0.99) { OnlyWhen = new Chosen("optimizer", "sgd") }
keep the test rows still across deals .SplitAtRandom(0.70, 0.15, seed: 100 + deal, testSeed: 7)
run it study.Run(deal => Pdd.Create()…Build())
take the winner found.Best.Declared · found.Winner
stop a trial that is not going anywhere Pruner = new MedianPruner(startup: 5, warmUp: 1).Prunes
go on from an earlier study study.Run(pipelineFor, earlier)
watch it, and stop it OnTrial = … · study.Run(pipelineFor, token)
compare two pipelines new Comparison(network => …) { Deals = 10 }.Run(first, second)

The rest of this page explains each one.

A study

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var space = new SearchSpace(
[
    new LogRange("rate", 1e-3, 1e-1),             // spread across the decades, as a learning rate is
    new WholeRange("units", 4, 32),
    new Choices("activation", ["relu", "tanh"]),
]);

var study = new Study(space, (candidate, network) =>
{
    var layered = network.Dense(candidate.Whole("units"));
    layered = candidate.Choice("activation") == "relu" ? layered.Relu() : layered.Tanh();

    return layered.Dense(1).Adam(candidate.Number("rate")).BinaryCrossEntropy().Run(seed: 20260929, epochs: 10);
})
{
    Trials = 12,
    Deals = 3,
    Sampler = new RandomSampler(11),
};

var found = study.Run(deal => Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitAtRandom(0.70, 0.15, seed: 100 + deal, testSeed: 7)       // another deal of train and validation, the same test rows
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", Scale.MidRange)
    .Normalise("fare", Scale.MidRange)
    .Normalise("sibsp", Scale.MidRange)
    .Normalise("parch", Scale.MidRange)
    .Report(report => report.Measure(Metric.Accuracy).On(Part.Validation, Part.Test).As(Shown.Numbers))
    .Build());

foreach (var trial in found.Trials)
{
    Console.WriteLine($"{trial.Number,2}  {trial.Score:0.000} ± {trial.Spread:0.000}  rate {trial.Candidate.Number("rate"):0.0000}, {trial.Candidate.Whole("units")} units");
}

var best = found.Best.Declared;                                    // the winner, as a step a pipeline keeps in its file
var onTest = found.Winner[0].Measures!.Parts.Single(part => part.Part == Part.Test);

Console.WriteLine($"{best.Epochs} epochs, accuracy {onTest.Values[0].Value:0.000} on {onTest.Rows} rows nobody chose by");

The space, and the network a candidate declares

A candidate is a point of a space, and the network it declares is written by you. The space names what may vary:

Dimension What it draws
NumberRange a number between two bounds
LogRange a number that spreads across decades
WholeRange a whole number
Choices one of a few words

The lambda turns the values a trial drew into the network, in the same chain .WithTorch(…) uses.

Anything a chain can say can vary

The lambda declares the network in PyTorch's words unless the study's Words says Keras's. So what 0.10.0 added is a point of the space like any other number: a penalty on a layer's weights, a weight decay, whether a layer has a bias, how many batches go into a step.

using DeepSharp.Learners.Networks;

var space = new SearchSpace(
[
    new LogRange("l2", 1e-6, 1e-2),
    new LogRange("decay", 1e-6, 1e-2),
    new WholeRange("every", 1, 8),
    new Choices("bias", ["with", "without"]),
]);

var study = new Study(space, (candidate, network) => network
    .Dense(16, bias: candidate.Choice("bias") == "with").L2(candidate.Number("l2")).Relu()   // a penalty on this layer's weights
    .Dense(1)
    .Sgd(0.05, momentum: 0.9).WeightDecay(candidate.Number("decay"))                       // and a decay on every number
    .BinaryCrossEntropy()
    .StepEvery(candidate.Whole("every"))                                                   // a step from so many batches
    .Run(seed: 20260929, epochs: 10));

A setting drawn only when another is chosen

A dimension's OnlyWhen names the choice of another dimension it is drawn under. new NumberRange("momentum", 0, 0.99) { OnlyWhen = new Chosen("optimizer", "sgd") } draws a momentum only for a trial that drew Sgd. In any other trial the dimension has no setting at all, which the lambda finds among candidate.Settings.

A dimension names one declared before it, or the space is refused where it is made.

What the winner's file keeps

What a trial chose at the meaning of leaving it out is not written into the winner's declaration, so a winner that kept the defaults is the file it always was.

The shape is yours

Nothing about the network's shape is fixed by the study: a layer you add when units is above sixteen is only code. A network of your own, written as a step, is a candidate too: give the study a function from a candidate to a LearnNetworkStep.

Validation chooses; the test rows are for the winner alone

Each candidate's score is its validation loss: the mean of the deals when there are several, with how much it moved from one deal to the next.

  • The study never trains on the validation rows.
  • It never reads a trial's test rows: a trial keeps no network.
  • Only the winner's networks — found.Winner, one for each deal — are kept, with the report's measures of the test rows.

A study that chose by the test rows, even once, would have measured a number it had picked the best of, and a best of twelve is flattered.

Deals: another division, the same test rows

A deal is another division of the rows into training and validation, and the test rows must not move.

One division cannot separate a better network from a luckier split: the same network scores a few percent differently on two divisions of the same rows. So the study can prepare the pipeline again, dealt another way, for each deal, and judge a candidate by the mean over them.

But a split at random deals every part from one seed, so a second seed would also deal another test part, and a candidate would be chosen on rows another deal learned from. SplitAtRandom(train, validation, seed, testSeed) fixes the test rows with testSeed and lets seed deal the rest. With more than one deal, a study whose deals do not share one test part is refused, saying so.

Stopping a trial that is not going anywhere

Pruner is handed the Progress of each deal as it ends — the deal, the scores so far and the trials before — and stops the trial when it says so.

MedianPruner(startup: 5, warmUp: 1) is Optuna's rule: once five trials have finished, a trial whose score so far is worse than the median of theirs at the same deal, after its first deal, is stopped.

  • A pruned trial says Pruned, keeps what it heard in Heard, and is never the best.
  • Its unit is the deal, the score a trial is chosen by, and never a loss halfway through an epoch, which is not the loss the run ends at.
  • A pruner is refused beside a Score of your own, since a median of a score that can mean anything decides nothing.

Going on from the trials run before

Run(pipelineFor, earlier) takes the trials of a study that stopped and runs the rest. Their numbering, their candidates and the space they were drawn from are held to this one's, and refused by name otherwise.

The best of them is trained again on every deal, and refused by name if its scores are not the ones it was handed. So the winner a resumed study gives back is a network it trained itself.

Watching a study, and stopping it

A study that takes hours can be watched and stopped.

  • OnTrial is handed every trial as it ends, in order, with how it was judged. Hearing the trials changes no number.
  • Run(pipelineFor, token) stops before the next trial, and a trial before its next batch, by throwing an OperationCanceledException that carries the token. Nothing is returned, because a study cut short has not found what it set out to find.

The same study run again is the same numbers

Every number a trial draws is worked out from the sampler's seed and the trial's place. So trial five is the same candidate whether the study has ten trials or a hundred, and a dimension added to the space leaves the values of the others where they were. Trials run one after another and share nothing they change.

Random search ships; a smarter one plugs in

A study asks an ISampler for the candidate of each trial and hands it the trials that finished before. So a sampler that learns from them — a tree of Parzen estimators, say — can be written in any assembly and given to the study. It is handed no setting for a dimension not drawn under the choices drawn.

Random search is the baseline a cleverer one has to beat. Whether a tree of Parzen estimators beats it depends on how many trials there are: over a few dozen it often does not, over a hundred it usually does.

The study chooses by finished networks alone; a pruned trial is never chosen.

Comparing a second pipeline with the first, in pairs

Whether a change helped is the same question as which network is best, one level up, and it has the same trap: a difference measured on one division is as large as the noise of the division. A Comparison trains one network behind two pipelines on each of many deals and reports, deal by deal, how far the second sits from the first.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

Pipeline Passengers(int deal, bool stratified)
{
    var declared = Pdd.Create()
        .ReadCsv("titanic.csv")
        .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("sex").Optional("age", ColumnKind.Number).Number("fare"));

    var split = stratified
        ? declared.SplitStratified("survived", 0.70, 0.15, seed: 100 + deal)
        : declared.SplitAtRandom(0.70, 0.15, seed: 100 + deal);

    return split
        .Target("survived")
        .FillMissing("age", With.Median)
        .EncodeCategories()
        .Normalise("age", Scale.MidRange)
        .Normalise("fare", Scale.MidRange)
        .Normalise("sibsp", Scale.MidRange)
        .Normalise("parch", Scale.MidRange)
        .Build();
}

var comparison = new Comparison(network => network.Dense(8).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 1, epochs: 10))
{
    Deals = 10,
};

var effect = comparison.Run(deal => Passengers(deal, stratified: false), deal => Passengers(deal, stratified: true));

Console.WriteLine($"{effect.Mean:0.0000} ± {effect.Spread:0.0000} over {effect.Deals} deals (standard error {effect.StandardError:0.0000})");

On the passenger list the tutorial reads, that prints a mean of −0.0069 with a spread of 0.0378 over ten deals: whether the share of survivors is kept the same in every part changes nothing a network of this size can show.

Reading the difference

The difference is the second pipeline's score less the first's, so with a loss for a score a positive mean says the second is worse. It is read against its own spread:

  • a mean of 0.01 with a spread of 0.05 is a deal's worth of luck;
  • a mean of 0.01 with a spread of 0.002 is a difference.

The two pipelines may differ in anything — a feature left out, a gap filled another way, a split at random against a split by time. A change that helps on one split and hurts on another shows as a spread larger than its mean, which is what a feature that knows the future of the rows it is measured on does.

Next: Networks has the layers, the optimizers and the loop a candidate is made of · Splitting the rows has the test seed