🧱 Optical Flow - articulateinstruments/AAA-DeepLabCut-Resources GitHub Wiki

Optical flow is intended to indicate the frame-to-frame shift in pixel direction and so might be expected to indicate internal as well as surface tissue movement. There are two ways to display this movement, either by vectors or by colour.

^{In practice this function is not considered to be very useful as speckle can be generated by internal reflections. Speckle artefacts of this type do not directly correspond to movement of internal tissue structure. Optical flow has therefore not been extended to allow the export or analysis of flow values. Please contact Articulate Instruments if extension of this function is desired.}

The Optical flow Edit dialog provides control of the algorithm.

The algorithm works as follows:

• The analysis is not performed on the interpolated image but instead it operates on the matrix of raw ultrasonic scanline data of size Scanlines x PixelsPerScanline. Typically with the Micro this is a 64 x 946 matrix of pixel values.
• Because the number of pixels along the scanlines is 14.7 times the number of scanlines, it is recommended to decimate the number of Y pixels by 15 so that the image is represented by a 64 x 63 matrix of pixels. i.e. set Subsample 1 in N to 15. This also speeds up the calculation.
• Scanlines are referred to as “Fan Vectors” in the dialog. The size of the patch is defined by Patch Width x Patch Height. A 5 x 5 patch is recommended.
• A patch in the previous frame is then correlated with all the patches in the current frame within the range of relative positions Patch Search Offset (x) and PatchSet Offset (y). In the example below the patch is 3x3 and the offset shown is (2,-3). The the "best" patch is the one with the...
  • Best Correlation – Finds the offset with the highest correlation and sets the flow vector according to the direction and distance of this offset.
  • Mean Correlation – Sums vectors for all possible offsets with magnitude defined by their correlation values, then divides by the sum of all the correlation values to find the direction and magnitude of the mean of all possible offsets.
  • Mean Covariance – As previous but using covariance rather than correlation.

The vector period sets how densely the arrows/colour blocks are plotted. Set vector period = 1 for best coverage. Set vector period to a higher value for sparser drawing of arrows.

The approach implemented here doesn't deal with the problem of edges. If you have a moving edge, correlation will only show movement perpendicular to the edge. It won’t indicate movement parallel to the edge.

In the image below with the settings in the dialog example image above it shows yellow indicating anterior tongue is lowering and purple indicating the posterior tongue and hyoid are retracting. Arguably, short tendon to tongue keypoint distance measures show this more clearly and measurably than optical flow.