GSoC 2021 Final report - orcasound/orca-embeddings GitHub Wiki

This Gsoc project is mainly an effort to improve the current sampling on the Orcal tool as is stated on this issue but ended up with a strong part of data visualization and embedding experimentation. Three points of the issue were addressed

  • Add new sampling strategies to Orcaal, and analyze performance.
  • Add Tensorboard visualizations to Orcaal for ML experts
  • Experiment with the full Orcasound stream

First Phase

As the new sampling strategies relied on embedding extraction the First phase was about building a pipeline extraction for different models hosted on Tensorflow hub, testing on several audio data and draw conclusions for the different embedding models. The first phase is condensed on this repo

Code Contributions

  • Embedding extraction function with the required preprocessing for 3 models Vggish, Yamnet and humpack whale. bf4497, 4a7bd4
  • Experiments with embeddings on continuous audio snippets and discrete clips Notebook
  • Tensorboard projector tutorial to publish experiments with audio data Notebook
  • Embedding projector examples on the orcaml repo data with the 3 models. b6a688
  • Performance test on embedding extraction in a GPU/non-GPU environments with different audio lengths bbc1d4, Notebook
  • Similarity Query examples using extracted embeddings Notebook
  • Added function to create animations from audios to see the time evolution of the embedding together with the signal and the spectrogram representation 2114b8

Second Phase

After the embedding experimentation phase, the second phase was about integrating the workflows tested into the Orcaal tool, All of the work is on the embeddings branch

Code contributions

  • Integrate embedding extraction on the current preprocess pipeline 6cc23f, 5403d3
  • Added function to compute distances, similarity and representativeness from extracted embeddings 6f4bd27
  • Added functions to test the app locally bb1467
  • Added information metric to data returned in predict 410fad
  • Generate files to publish embedding projector from preprocessed folders e07d1c
  • Analyze sampling strategies with embedding visualizations and clip listeninga106cfc

To do

  • Add the new information metric into the postgres database.
  • Add endpoint to use the alternative metric.

Blog posts