Neighbor Masking - kristinbranson/APT GitHub Wiki

Neighbor Masking is a preprocessing step taken where targets close to a given target ("neighbors") are "erased" so that the given/central target can be tracked and analyzed as if it were alone. Pose-trackers may find it easier to analyze a lone target as compared to a target surrounded by many neighbors.

Use of Neighbor Masking requires that a project:

  1. Contain trajectories (trx) giving positions of target animals. The trx fields .x, .y, .theta, .a, .b are used/required depending on the method selected.
  2. Supplies a function that reads/generates a background image for a given movie.

In the above before/after pair of images, a neighboring target is erased via Neighbor Masking.

To perform Neighbor Masking, APT performs the following steps. User-configurable parameters are available under Track>Configure Tracking Parameters, Multiple Targets>Mask Neighbors.

  1. The user-specified Background Read function is called for the movie in question. This returns i) a background image and ii) a background uncertainty/deviation image. The results of this call are somewhat sensibly cached. The Background Read Function must return an unchanging, constant value for a given movie.
  2. For the frame in question, a background subtraction is performed per the user-specified Background Type. The result is a difference image.
  3. A single user-specified Foreground Threshold is applied to the difference image to generate a black/white foreground label image.
  4. Pixels labeled as foreground are assigned to targets according to MaskNeighbors>Method:
    • Connected Components. Connected components in the foreground label image that contain a trx center (trx.x,trx.y) are assigned to that target. Connected components that contain two or more trx centers are not split. This method will not work well when targets are frequently touching in frames of interest as touching neighbors will not be masked.
    • Gaussian Mixture Model. An Expectation-Maximization algorithm is used to determine an optimal mixture of Gaussians to cover the foreground label image, using the difference image as weights. There is one Gaussian for each target/trajectory and each Gaussian is initialized using trx.a, .b, and .theta.
    • Empirical PDF. All labeled frames containing only a single target are used to generate an empirical distribution indicating how likely it is that a given location/pixel is a foreground pixel relative to that target. When two or more targets are in proximity to each other, foreground pixels in the vicinity are assigned to the target that maximizes this likelihood.
      • To use this method, you must call the method lObj.updateFGEmpiricalPDF() first to generate and store the empirical PDF for your project. This method can be called at any time to refresh/update this PDF, stored in the Labeler property .fgEmpiricalPDF. You are wholly responsible for updating this property/PDF as your project evolves. The empirical PDF is not automatically updated when you add or remove labeled frames, change the foreground threshold, etc. You may want to use a single "model" PDF for your organism/rig, across multiple projects for example.
      • To be clear, the generated empirical PDF depends on the background images, foreground threshold parameter, etc that are set/used at the time updateFGEmpiricalPDF is called. If any of these inputs change, you might want to update your PDF accordingly. Warnings are thrown in some cases.
  5. The initial segmentation may leave "orphaned" foreground components that are assigned to a target yet have no connected path back to that target's trx center. To remediate this, there is a second segmentation pass that reassigns any dangling or orphaned components.
    • One can imagine a variety of approaches to this second pass. Currently a very simple heuristic is applied wherein an orphan is reassigned to an arbitrary target that borders directly on the orphan.
  6. Segmentation is now complete; all foreground pixels are assigned to a target. (In the case of 'Connected Components', touching targets may be assigned a single label/connected component). All pixels not assigned to the target of interest are replaced with corresponding pixels taken from the background image. This concludes the Neighbor-Masking operation.