Step 1 — Reading a file

In short

The first step of the course names where the rows come from. Nothing is decided here and nothing is learned: the reader hands every cell over as text, and the next step says what each column holds. The one rule to remember: an empty cell arrives as an absence, and it stays one until a later step settles or fills it.

You want to…	Write
read a comma-separated file	Pdd.Create().ReadCsv("titanic.csv")
see a proposed kind for every column before writing the schema	.ReadCsv("titanic.csv").ProposedKinds()
read a Parquet file, an Excel workbook or a JSON file	.ReadParquet(path), .ReadExcel(path), .ReadJson(path) — each from a package of its own
read a folder of texts, pictures or recordings, a folder to a class	.ReadTexts(path), .ReadImages(path, new ImageShape(28, 28, 1)), .ReadSounds(path, new SoundShape(16000, 16000, 1)) — pictures and recordings each from a package of their own
read two files as one source, joined on a key	.ReadJoin("planned.csv", "arrived.csv", ["flock"], Unmatched.Refuse)
leave out a left row that has no partner, instead of stopping	Unmatched.Drop in place of Unmatched.Refuse

The rest of this page explains each one.

What the step looks like

using DeepSharp.Pipelines;

var proposal = Pdd.Create()
    .ReadCsv("titanic.csv")
    .ProposedKinds();                      // every cell read, a kind proposed for each column

foreach (var column in proposal.Columns)
{
    Console.WriteLine($"{column.Name}: {column.Kind}");
}

A proposal, not a schema

ProposedKinds() is there because a schema typed from nothing is a schema typed from memory. It reads the file and proposes a kind for each column — alive true or false, embarked a category, age a number.

It is a proposal and nothing more. You write the schema; the proposal saves you from opening the file in a spreadsheet first.

Other kinds of file

A comma-separated file is one reader of several. Parquet, Excel workbooks and JSON each have their own, in a package of their own, so a project that reads only CSV carries only CSV.

Which readers exist, and what each one does with a column it cannot make sense of, is on Pipeline.

A folder to a class

Pictures, recordings and texts are kept the way they are learned from: one folder to a class, one file to an example. A file is a row, the rows come in the order of the files' paths, and the name of the folder a file stands in is its answer — as a column label, and as a column of nought and one for each class, which Labels takes. A file lying beside the folders belongs to no class and is refused.

The step says the shape every picture or recording is read as, and one that is another shape is refused, naming the file and both shapes. Nothing is resized, cut, mixed down or resampled, because the pipeline's file would then have no word for a thing that was done to the data. A row is known by what its file holds and not by where the file lies, so renaming a file or moving the folder changes no key. The notebook has no block for these readers, since every view would read every file again: read the folder in a C# cell and hand the rows over.

Two files can be one source

Planned values in one file and what arrived in another are joined where the pipeline can see it, not by hand before it starts. .ReadJoin("planned.csv", "arrived.csv", ["flock"], Unmatched.Refuse) gives each row of the left file beside the row of the right file its key names.

The rules of a join:

• the key is the exact text of the columns named;
• many left rows may share one key;
• a key the right file holds twice is refused;
• what becomes of a left row with no partner is said, never assumed.

Pipeline has the rules.

What the row looks like after this step

The passenger whose age the file leaves empty arrives as an empty cell — not a nought, not a zero-length word, an absence. It stays an absence until something settles or fills it, and every step between here and there can see that it is one.

Next: Declaring the columns · Every reader and its options: Pipeline