Skip to content

Running ZeroCostDL4Mic notebooks on FloydHub

lucpaul edited this page Dec 22, 2020 · 3 revisions

The notebooks in the ZeroCostDL4Mic project can be exported to other platforms that can use jupyter notebooks. This means that users who encounter resource limitations in Google Colab can easily use the notebooks in more powerful cloud-computing platforms. We had good experience with importing the notebooks to the platform FloydHub https://www.floydhub.com/ which provides services beyond those available in Google Colab, but for a fee which is calculated based on GPU usage.

Below is a step-by-step guide on how to import and use the ZeroCostDL4Mic notebooks in FloydHub (instructions based on StarDist 2D notebook).

  1. After Creating a FloydHub account, create a new project
  2. Create a new workspace
  3. In the pop-up window, click on import Github repository, and paste the link to the ZeroCostDL4Mic repository. This will make all the notebooks in the repository immediately available in the FloydHub workspace.
  4. Choose an environment. For StarDist 2D, use tensorflow 1.15.2. For other notebooks based on Tensorflow, this is also the recommended version.
  5. Choose a machine, either CPU or GPU, depending on the requirements for training. While CPU training is free, FloydHub charges the user for GPU use by the hour.
  6. Get your training dataset:
  • if using the dataset from this project the data can be downloaded via the FloydHub terminal plugin by typing the following command to download the dataset: wget zenodo.org/record/dataset_ID_number/files/dataset_name.zip where dataset_ID_number is the unique 7-digit zenodo ID number of the dataset (also in the URL) and dataset_name is the name of the folder containing the dataset, for example ‘Stardist_v2’ for the Stardist dataset. The user can then unzip the folder by typing: unzip dataset_name.zip
  • to upload their own dataset the users need to follow the instructions provided by FloydHub, found here.
  1. After these steps, the github repository and the dataset folder are available in the created workspace. Any notebook can be opened from the left-hand menu and displays the same interface as in colab, with the exception of showing the code cells permanently, without the Colab graphical user-interrface.
  2. To use the notebooks, users may need to make a few modifications to the notebooks. The main modifications, valid for most notebooks:
  • comment out the %tensorflow_version 1.x magic command in cell 2 by prepending #
  • the from __future__ import … command was copied to the top of cell 2
  • astropy needed to be installed before import by adding !pip install astropy to cell 2

Additional modifications

Further changes to the notebook simplify the experience of use on this platform, e.g. deleting the string ‘/content’ in pathnames or replacing it with ‘/floyd/home’ where it occurs in the notebook. This is necessary because the ‘content’ folder is specific to colab which will result in an error when using the notebook in FloydHub. Additionally, copying paths from the left-hand menu omits the root folder ‘/floyd/’ which we needed to add manually when pasting a dataset or model path into the notebook.

Different modifications may be necessary for different notebooks. To make the necessary modifications, it is advised to play the cells in FloydHub and update the code after reading the error-messages displayed in the output. The most likely errors will relate to packages missing from FloydHub (in this case add !pip install missing_package and import missing_package to the import dependencies packages cell of the notebook) or path names including 'content' (see above).