Developer documentation - nimwegenLab/moma GitHub Wiki

WARNING: THIS IS A DRAFT VERSION. IT IS NOT YET READY FOR USE.

• TODO: Merge with Developing-MoMA.md

This page describes how to get started with developing MoMA code. It contains instructions for setting up the build environment, checking out the code with Git, and building the code.

For instructions on how start using MoMA for analyzing your data please see the MoMA tutorial introduction instead.

MoMA was developed using IntelliJ IDEA, which is a Java IDE. The professional edition is free for academic use. The community edition is free for all uses.

You can find instructions on how to install IntelliJ IDEA here. Development was done exclusively under Linux, so this the recommended OS for development. However, IntelliJ IDEA is also available for Windows and MacOS.

You should also be able use Eclipse IDE for development, but it is untested.

You require a Gurobi license and installation in order to develop and use MoMA. Moreover, different version of MoMA use different versions of Gurobi (the Gurobi dependency was updated with new MoMA versions). The page Gurobi setup instructions provides information how to obtain a license, which licenses are available, and how to install Gurobi.

The final ~/.bashrc entry for setting up Gurobi will then look like this, where you need to replace /path/to/my/gurobi_license.lic with the path to your license file:

export GRB_LICENSE_FILE="/path/to/my/gurobi_license.lic"
export GUROBI_HOME="/opt/gurobi1003/linux64"
export PATH="${PATH}:${GUROBI_HOME}/bin"
export LD_LIBRARY_PATH="${GUROBI_HOME}/lib"

1. Clone the MoMA repository:

   git clone https://github.com/nimwegenLab/moma.git

   This creates the directory moma in the current directory.

2. Open IntelliJ IDEA and select Import Project. Navigate to the moma directory and follow the instructions for import.

3. You can now build MoMA from within IntelliJ IDEA by selecting in the menu: Build -> Build Project (shortcut: ctrl+F9).

You can import the MoMA repository directly from inside IntelliJ IDEA. The following screenshots show the steps to do this:

Select "Project from Version Control..."	Enter Git repo URL and target project folder

To develop MoMA you need to download the deep-learning segmentation model. The model is not included in the MoMA repository. You can download the model here. Furthermore, you will need to set the path to this zip-file in the mm.properties file of the test-dataset in the next step.

TODO: ADD URL TO TEST DATASET

Download the developer test data here. It contains two files:

• A TIFF file containing a time-lapse recording of a single growth-lane
• A mm.properties config-file containing the settings used for tracking the growth-lane.

After downloading and extracting the data you will need to set SEGMENTATION_MODEL_PATH in mm.properties to the location of the U-Net model that you downloaded in the previous step:

SEGMENTATION_MODEL_PATH=/home/micha/.moma/unet_models/model_20210715_5b27d7aa/current_tensorflow_model.zip

NOTE: The mm.properties file also contains the path IMPORT_PATH of the TIFF file that is being tracked. You do not need to change this value. During testing you will call MoMA with the correct path to the TIFF file, which will then be stored to the final mm.properties on export.

There exist two release branches:

• release/v0.9: Current stable release branch. The master branch currently points to this branch.
• release/v0.10: Release with the addition of an entry-assignment, which allows new cells to enter a growth-lane at the top.

Since release/v0.10 has not been tested much, it is recommended to continue development on release/v0.9 and port changes to release/v0.10 as needed.

To checkout the master branch run:

git checkout master

The repository includes this script in its root directory: start_topic_branch.sh. It allows you to quickly setup a new topic branch for development, along with sample-data (you need to configure this) and a test-class.

You need to edit these paths in the script to suite your environment:

devel_data_folder="<DEVELOPMENT DATA DIRECTORY>"
template_class_path="<MOMA GIT REPOSITORY ROOT>/src/test/java/com/jug/TEMPLATE_CLASS_FOR_INTERACTIVE_TESTING.java"
image_file_name="20211026_VNG1040_AB6min_2h_1_MMStack_Pos7_GL12.tif"
config_file_name="mm.properties"

# [ elided code ]

test_folder_path="<MOMA GIT REPOSITORY ROOT>/src/test/java/com/jug"

# [ elided code ]

topic_data_template_folder="$devel_data_folder/<SUBFOLDER WITH YOUR TEMPLATE DATA>"

where:

• <DEVELOPMENT DATA DIRECTORY>: The directory where you want to store the test data. You will need to create this directory.
• <MOMA GIT REPOSITORY ROOT>: The path to the root-directory of the moma Git repository.
• <SUBFOLDER WITH YOUR TEMPLATE DATA>: Subdirectory that contains your template data that is used to create the data-directory for the test.

You can now, run e.g. the command ./start_topic_branch.sh feature 20230930-my-new-feature and confirming the prompts will create the following output:

$ ./start_topic_branch.sh feature 20230930-my-new-feature
The full topic branch name is:
        feature/20230930-my-new-feature
Continue? (y/n)y

Branching from current branch:
        release/v0.9
Continue? (y/n)y

Session type:
        feature
Session name:
        20230930-my-new-feature
Starting feature session on branch:
        feature/20230930-my-new-feature
The data folder for this session is:
        /home/micha/Documents/01_work/15_moma_notes/02_moma_development/feature/20230930-my-new-feature
The class for this session is:
        /home/micha/Documents/01_work/git/MoMA/src/test/java/com/jug/feature/Feature__20230930_my_new_feature.java
Switched to branch 'feature/20230930-my-new-feature'
[feature/20230930-my-new-feature 20356410] "Started new feature-session. Commiting test-class for this session: Feature__20230930_my_new_feature.java"
 1 file changed, 62 insertions(+)
 create mode 100644 src/test/java/com/jug/feature/Feature__20230930_my_new_feature.java

This is an example-excerpt from the test-class that was generated by the previous command, which contains methods for calling MoMA in different modes with the provided test-data:

// [elided code]

    public static void main(String[] args) {
        Feature__20230930_my_new_feature tests = new Feature__20230930_my_new_feature();

        tests.run_interactive();
    }

// [elided code]

    public void run_interactive() {
        Path inputPath = Paths.get(datasetsBasePath, datasetSubfolder, "20211026_VNG1040_AB6min_2h_1_MMStack_Pos7_GL12.tif");
        Path properties_file_path = Paths.get(datasetsBasePath, datasetSubfolder, "mm.properties");
        analysisName = "test_interactive";
        startMoma(false, inputPath.toString(), null, tmin, tmax, false, new String[]{"-f", "-p", properties_file_path.toString(), "-analysis", analysisName});
    }

// [elided code]

Supported arguments for topic-branches are:

• feature: branch to develop a new feature
• bugfix: branch to fix a bug
• exploration: branch to explore e.g. a new feature/functionality
• cleanup: branch to cleanup code; e.g.: refactor or remove unused code

You will need to set the path to the Gurobi license file in the run configuration for the test-class that you generated in the last section. To do this:

• Select "Edit configurations...":
• Ensure your test-class is selected in left panel of Configurations window and select "Edit environment variables":
• Add the environment variable GRB_LICENSE_FILE to point to the location of your license file:

You may get the following error during the build:

java: Compilation failed: internal java compiler error

The IntelliJ IDEA build output looks like this:

The reason is that for some branches the SDK was set correctly incorrectly (it was automatically set to Python while working on the Docker image).

To fix this, go to File -> Project Structure -> Project and set the Project SDK to the correct Java SDK:

Select JAVA SDK and select OK	Rebuild project

This describes how to change or update the Gurobi version used by MoMA.

Gurobi LLC does not provide a Maven packages on the Maven central repository before Gurobi versions <10.0.3. Therefore the Gurobi dependency must be copied to a local repository in order to include it during the build. The following steps need to be followed to change this version:

1. Download the desired version (here we assume version 10.0.0 for Linux): https://packages.gurobi.com/10.0/gurobi10.0.2_linux64.tar.gz

2. Copy gurobi10.0.2_linux64.tar.gz to /opt/ and run:

   tar xvfz gurobi10.0.2_linux64.tar.gz

   For further instructions see the Gurobi installation guide here.

3. Copy gurobi10.0.2_linux64/gurobi1002/linux64/lib/gurobi.jar to lib/gurobi.jar in the MoMA Git repository. Overwrite the previous gurobi.jar.

4. In pom.xml update the following entry to use 10.0.2:

         <dependency>
               <groupId>gurobi</groupId>
               <artifactId>gurobi-jar</artifactId>
               <version>10.0.2</version>
               <scope>compile</scope>
           </dependency>

5. In deploy.sh change: GRB_VERSION="10.0.2"

6. In Dockerfile change:

   ARG GRB_VERSION=10.0.2
   ARG GRB_SHORT_VERSION=10.0

WARNING: Work in progress. This section is not yet complete. This section needs to describe how one can create a release using Docker and DockerHub.

1. Create Git tag with annotated tag with version tag on the commit of interest: git tag -a v0.9.9. Annotated tags keep a history of who created that tag and you must write a commit message, where I usually just wrote "Bump version to vX.X.X" (https://git-scm.com/book/en/v2/Git-Basics-Tagging).

• Tag name is v<Major>.<Minor>.<Bugfix>.
• Push this tag to Github when ready (i.e. after testing), e.g.: git push origin tag v0.9.9 for linking the commit and docker-image for future reference.

2. Build docker image using: docker/build_docker_image.sh
3. Push docker image to docker hub using: docker/push_to_dockerhub.sh

• You will have to put in the credentials (the username is: nimwegenlab and the current contact email is [email protected]).

Setup instructions:

• To setup TensorFlow GPU support follow the instructions here: https://www.tensorflow.org/install/gpu
• Add /usr/local/cuda/lib64 to LD_LIBRARY_PATH in the configuration of the run/debug configuration in IntelliJ IDEA for your test.

Note on testing code with TensorFlow GPU:

• Running TensorFlow on GPU outputs slightly different numerical results for predictions from CPU. This is not a problem for the MoMA analysis, but it is a problem for some tests, which assert the output against reference data that was produced using TensorFlow on CPU and do not consider numeric imprecision. Therefore, it is recommended to not use TensorFlow on GPU during development or when running tests.