Global Workflow - Audiveris/omr-dataset-tools GitHub Wiki

The following diagram presents a global scenario for building and using the OMR Dataset.

Dataset Workflow

Task #1. Populating the OMR dataset

The OMR dataset is meant to be a vast collection of pairs "image+annotations", where the annotations describe every symbol, with its shape and precise bounds, as found in the sheet image.

The diagram depicts several ways to populate the dataset:

  • Using MuseScore symbolic data (.mscz files) to generate synthetic images with their annotations,

  • Using specific tools to add noise / distortion to synthetic images,

  • Using Audiveris OMR data (.omr) to extract image and related annotations.

Task #2. Training a classifier

A deep neural network, when properly trained, should be able to work directly on the pixels around a given location in a sheet image and tell which symbol, if any, is present there. The first and foremost role of the OMR dataset is to provide the needed material for such training.

For each fixed-shape symbol in the dataset, we simply need all the pixels located in a rectangular window centered on the symbol area. The image must be scaled to a predefined interline value and all symbol windows have the same rectangular size.

There are 3 steps in this training task:

  1. Features

    • Add one row into output sheets.csv file, whose format is: sheetId, sheetPath,
    • If needed, scale the original image to predefined interline,
    • Optionally, generate some none symbols beside the valid symbols,
    • Extract the pixels from each (valid or not) symbol window as a flat vector,
    • Store the vector and symbol index as one row of output file features.csv,
    • Store meta-data as one row of journal.csv, whose format is: row, sheetId, symbolId, interline, x, y, w, h, shapeId,
    • Store mean values and standard deviations for pixels and symbol dimensions into corresponding pixels.dat and dims.dat output files,
    • Optionally for visual inspection, generate a control image with valid symbol bounds and invalid locations.

  2. SubImages (optional step, meant for visual checking)

    • For each row of features.csv, regenerate the corresponding sub-image.

  3. Training

    • Allocate a new deep neural network (or reload it, if it already exists),
    • Read batches of vectors out of features.csv,
    • Pre-process each vector to normalize pixel values with pixels.dat,
    • Iterate training on batch,
    • Optionally for visual inspection, trace mistaken symbols back to their original data thanks to meta-data found in journal.csv and sheets.csv,
    • Finally store the network trained parameters with pixel normalizer into img-classifier.zip.

Task #3. Using the trained classifier

Once we have a trained classifier, we can use it within an OMR engine like Audiveris:

  • Reload the classifier and pixel normalizer from img-classifier.zip,
  • Take a view, properly scaled on predefined interline value, of a music sheet,
  • Normalize its pixels by applying the pixel normalizer,
  • For any given location, extract the vector of pixels out of fixed rectangular window,
  • And run the classifier on the vector, to get the most probable symbol(s) centered on the location.