3D Volumetric Instances to Mesh steps - JamesDarby345/Volumetric_Instance_to_Mesh GitHub Wiki

Purpose

This code converts volumetrically labelled cubes into .obj files for flattening and rendering. The process involves taking an instance label, reducing it to a midline label (a 1 voxel thick representation), and then using that label to create an .obj file. The code also aims to maintain/create the manifold and non-intersecting mesh properties required for the autosegmentation grand prize from the volumetric instance labels.

Steps

  1. Change the paths in the scripts to point to the 3d volumetric instance nrrd file folder, the expected name is zyx_mask.nrrd as on the download server, within the zyx folder names. Ex: 02000_02000_02000/02000_02000_02000_mask.nrrd
  2. Run filter_and_reassign_labels.py to filter out small objects and reassign labels in the nrrd files, which will make running the next scripts faster.
  3. Run a instance to midline volume conversion script, differences, pros and cons of each method are commented in the script and explained below. Update: Use fb_avg_3dinstance_to_midline_volume.py, it is the fastest and produces the best results now. By masking out voxels that left the original label followed by delauney2d, its downsides were negated.
  4. Run the midline_to_obj.py file to convert the midline volume to .obj files.

If you want to visualise the data after any of these steps, I recommend dragging it into Napari, or into 3D Slicer (hit the crosshair icon to center the view on the data in slicer).

All outputs will be saved in the target folder Ex: 02000_02000_02000/02000_02000_02000_fb_avg_mask_thinned.nrrd 02000_02000_02000/obj/02000_02000_02000_fb_avg_mask_thinned_1.obj

Instance to midline volume approaches

Front-back average

File: fb_avg_3d_instance_to_midline_volume.py

Update: Masking voxels that left the original label and relying on delauney_2d triangulation to fill holes results in this method being the fastest and produces the best results. Consider the other two deprecated. This approach also reduces the chances of mesh collision even further.

Update 2: Added a morph command line parameter, which defines a radius for morphological closing, this adds time to the computation but results in better, and denser midline volumes. Also changes to the parralelisation resulted in the code going from 3:36 to do 61 cubes to 0:39 on a M2 macbook pro without morphology.

Fastest midline calculation method. Front and back of the instance volume are averaged to create a midline volume. PCA components are used to determine the orientation of the structure. Doesn't natively fill holes. Disconnected fibres in the same label are an issue for this method as the front-back average ends up out in space off of the sheet. Follows midline more closely on aggressive curves. Morphological tunnels are filled during the mesh creation process. Still leads to some rougher label connections than the graph approach, but is more representative of the shape

Typical Result:

Outdated: Typical unideal/'failure' case: Note how the mesh is averaging into empty space, outside the sheet

New: Morphology Impact: Resulting midline volume without morphology on 'patchy' label:

Resulting midline volume with appropriate morphology value: This should allow for more uniform point cloud extraction and higher quality resulting meshes

Graph cut

File: graph_cut_3d_instance_to_midline_volume.py

Slow due to graph construction which adds valid seam/sheet prior. Results in 'smoother' midlines, but less accurate to the original midline volume. When the label changes directions aggressively, the midline can end up curving away from the original midline volume. Single value assumption along maximum PCA direction. Can only create voxels at maximum 45 degree angle to previous voxels, limiting ability to follow aggressive curves. Fill holes (morphological tunnels) well by using the graph construction to provide structure across the gap. Could cause obj collisions as midline labels can leave their instance label to allow for hole crossing, and end up inside of other objects.

Biggest drawbacks are compute time and potential for resulting mesh collisions Typical midline volume result: See how the volume is limited to 45 degree angles, which could result in the volume exiting the instance label in aggressive curves. Can be partially negated by running on smaller cubes/ROI's.

Distance map

File: dist_map_3d_instance_to_midline_volume.py

Faster than graph construction method, slower than front-back average method. Doesn't fill holes (morphological tunnels). Constructs distance map for each label Uses single value assumption along maximum PCA direction to take highest dist map value at each point to create the midline volume. Follows midline more closely on aggressive curves. Morphological tunnels are filled during the mesh creation process. Leads to closer midlines, but rougher labels, especially around holes. Should have a low chance of causing obj collisions as the midline cannot leave the instance label.

Typical unideal/'failure' case: See how the resulting mesh is connected between different parts of the sheet, resulting in rougher labels.

Discussion of Approach & Challenges

A key challenge is dealing with the holes, or more specifically the 'morphological tunnels' in the instance label. This is mostly dealt with by assuming the sheet can be represented accurately with a single value on each x,y line (1 per z-axis). The code also computes the PCA components and rotates the individual labels such that they face as perpendicular to the z-axis as possible to reduce the impact of this assumption. The midline volume is then calculated by one of the 3 approaches. The graph approach deals with morphological tunnels by using the valid seam constraint introduced by the graph construction. The other approaches rely on filling the holes during the mesh construction step. This requires setting a maximum connection length parameter, and could result in U-shaped meshes etc being filled as if it is a hole. Though unideal, this should have minimal effect on the downstream flattening and rendering task, just resulting in some empty space being rendered.