Pipeline Process - ActiveBrainAtlasPipeline/neuroglancer GitHub Wiki

Post CZI Creation Process

1. To install the software prerequisites, look here.. Or just use the installed virtual environment on ratto, basalis and muralis: source /usr/local/share/pipeline/bin/activate
2. Running any of these commands at high resolution, you will want to prefix each command with nohup and put a '&' at the end of the command. That will make it run in the background and you can log out.
3. Run: python create_meta.py --animal DKXX
   1. This will scan each czi file.
   2. Extracts the tif meta information and inserts into the slide_czi_to_tif.
   3. On basalis takes: 437m57.460s for 147 czi files.
   4. On muralis: 513m50.446s for 147 czi files.
   5. On ratto: 527m9.581s for 147 czi files.
4. Run: python create_tifs.py --animal DKXX --channel 1 --njobs <number_of_cores_to_use>
   1. This will read the sections view in the database. This can be run before the QC section below. Doing that will create some extra unused tifs, but that is not a problem.
   2. Create tif files for channel 1.
   3. Create png files for channel 1.
   4. Repeat process for the other 2 channels when ready.
   5. Muralis 40 jobs. This step is handled by java so parallelization is good.
5. Run tif2jp2.sh DKXX in programming/pipeline_utility/registration. This script runs Matlab so you must have a license. This will create jp2 files which CSHL uses.
6. Have someone confirm the status of each slide in: https://activebrainatlas.ucsd.edu/activebrainatlas/admin
   1. After logging in, go to the Slides link in the Brain category.
   2. Enter the animal name in the search box.
   3. Under the PREP ID column, click the link of the slide you want to edit.
   4. If the entire slide is bad, mark it as Bad under Slide Status.
   5. If a scene needs to be marked as Bad, Out of Focus or the End slide, select the appropriate Scene QC. If you mark it as Bad or Out of Focus, the nearest neighbor will NOT be inserted. Use the replicate tool to copy either the one before or the one after.
   6. If you want to replicate a scene, choose the Replicate field and add an amount to replicate.
   7. The list of scenes are listed near the bottom of the page.
   8. When you are done, click one of the Save buttons at the bottom of the page.
   9. After editing the slides, you can view the list of sections under the Sections link under Brain.
7. Run: python create_preps.py --animal DKXX --channel 1
   1. This will fill the directory of the full and low resolution files for channel 1.
   2. Repeat process for the other 2 channels when ready.
   3. View a couple thumbnails to determine how much rotation/flips to perform.
8. Run: python create_masks.py --animal DKXX
   1. This will read the thumbnail directory and create masks in the DKXX/preps/masks/thumbnail_colored dir. You will need to go into that directory and check each mask to make sure the mask is covering only the necessary parts. If there is either tissue missing or the mask is too big do:
      1. Use the black paintbrush to hide extra masking (the mask needs to be reduced)
      2. Use the white paintbrush to make the mask bigger.
      3. When you have checked all masks and they are good, rerun the create_mask.py program like this python src/create_masks.py --animal DKXX --final true This will create the black and white masks in DKXX/preps/masks/thumbnail_masked dir.
   2. No need to use channel 2 or 3. It works solely on channel 1.
9. Run: python create_histogram.py --animal DKXX --channel 1 --single single
   1. This will read the directory of the thumbnail resolution files for channel 1.
   2. Create histogram for each files for channel 1.
   3. Repeat process for the other 2 channels when ready.
10. Run: python create_clean.py --animal DKXX --channel 1
    1. The necessary rotation and flip parameters must be in the scan run table..
    2. Be careful with the rotation and flip. To do a 90 degree right rotation requires rotation=1
    3. View a few of the files in DKXX/preps/CH1/thumbnail_cleaned.
    4. full resolution on ratto one channel takes 863 minutes. When it is finished, check file count and make sure there are no small files, especially with high resolution.
11. Run: python create_elastix.py --animal DKXX
    1. This will create the DKXX/preps/elastix directory and a subdirectory for each file pair.
    2. The elastix dir will be used to align the other channels and the full resoltions.
12. Run: python create_alignment.py --animal DKXX --channel 1
    1. This will use the DKXX/preps/elastix directory for the transformation parameters. It will then use PIL to do an affine transformation to do section to section alignment.
13. Run: python create_web.py --animal DKXX
    1. This will use the DKXX/preps/CH1/thumbnail_aligned directory for source images. It will then use PIL to do an create web viewable images.
14. Run: python create_neuroglancer_image.py --animal DKXX --channel 1 --downsample true
    1. This will create the data for neuroglancer for channel 1.
    2. View results in neuroglancer. Add the layer to the precompute with: https://activebrainatlas.ucsd.edu/data/DKXX/neuroglancer_data/C1T
15. When you are satisfied with the results, run these steps with full resolution on channel 1
    1. python create_masks.py --animal DKXX --downsample false
    2. python create_clean.py --animal DKXX --channel 1 --downsample false
    3. python create_alignment.py --animal DKXX --channel 1 --downsample false
    4. python create_neuroglancer_image.py --animal DKXX --channel 1 --downsample false
    5. python create_downsampling.py --animal DKXX --channel 1 --downsample false
    6. View results in neuroglancer. Add the layer to the precompute with: https://activebrainatlas.ucsd.edu/data/DKXX/neuroglancer_data/C1
16. When you are satisfied with the full resolution results, finish the other two channels
    1. python create_clean.py --animal DKXX --channel 2 --downsample false
    2. python create_alignment.py --animal DKXX --channel 2 --downsample false
    3. python create_neuroglancer_image.py --animal DKXX --channel 2 --downsample false
    4. python create_downsampling.py --animal DKXX --channel 2 --downsample false
    5. View results in neuroglancer. Add the layer to the precompute with: https://activebrainatlas.ucsd.edu/data/DKXX/neuroglancer_data/C2
    6. python create_clean.py --animal DKXX --channel 3 --downsample false
    7. python create_alignment.py --animal DKXX --channel 3 --downsample false
    8. python create_neuroglancer_image.py --animal DKXX --channel 3 --downsample false
    9. python create_downsampling.py --animal DKXX --channel 3 --downsample false
    10. View results in neuroglancer. Add the layer to the precompute with: https://activebrainatlas.ucsd.edu/data/DKXX/neuroglancer_data/C3
17. To add masks to the mask training model do:
    1. Add 8 bit normalized files from a brain with for i in *.tif; do new=$(echo "DK62.$i"); cp -vi $i /net/birdstore/Active_Atlas_Data/data_root/brains_info/masks/normalized/$new; done
    2. Do the same with the thumbnail_masked dir
    3. Run the src/masking/train-mask.py file to create a new model in: /net/birdstore/Active_Atlas_Data/data_root/brains_info/masks/mask.model.pth