Generating false color vector field of matches - bootchk/resynthesizer GitHub Wiki

This documents a feature in the repository branch "falseColorMatch." Most users will have little practical use for this feature. Some developers may find a use for this feature in a sequence of image processing steps.

(The branch also contains, a Python language replacement plugin for the C language plugin resynthesizer-gui. That replacement is straightforward and poses little risk. The replacement is forward-looking by reducing a dependence on an old GTK library. The feature for false color match changes the core engine so little that the risk is very low. The branch is expected to be merged into the master branch.)

What it does

For each pixel, it returns a vector in the coordinate space of the source image, instead of a vector in color space.

Rendered as a color, the vector in coordinate space is a "false color" or "pseudo color." From the Wiki page on false color: "eases the detection of features which are not otherwise discernible."

The x coordinate is returned in the R pixelel, the y coordinate in the G pixelel, the B pixelel is alway 0.

How to use it

Start the Map>Resynthesize plugin and choose the "Generate false color vector field of matches" checkbox.

When you use it, you won't see the usual result. To also see the usual result, you will need to run the Resynthesizer twice. But also see: non-determinism.

All the other parameters have the usual effects on the core algorithm.

Motivation

The coordinates of the best match in the source are a more elemental result. The Resynthesizer has always computed said coordinates, but discarded them. More elemental: from said coordinates, one can recover the color of the best match in the source, but not vice versa. In other words, the color of a pixel in the usual result doesn't tell you exactly where it came from in the source.

As concrete examples of uses, I wondered whether the result could help detect objects or features in images. In other words, edge detection. Specifically, I hope to detect meteorite craters in images of farmland. I also wondered whether the result would help image forensics, e.g. to detect cloning in images.

The jury is still out whether the result has any practical use.

Coordinate scaling and loss of accuracy

In the interest of not breaking the API of the Resynthesizer engine, the engine returns the new result in the same data structure instance as the usual result. IOW, you get one or the other but not both.

The existing data structure for the result is 24-bit (eight bits for each of RGB.) Coordinates often require more than eight bits. Thus the result coordinates are always scaled (to a max of 255.) This causes a loss of accuracy.

When a dimension of the source image is smaller than 255, result coordinates are scaled up. When a dimension of the source image is greater than 255, result coordinates are scaled down. Integer arithmetic is used for scaling, and truncations occur.

Except for this loss of accuracy, scaling of the result is reversible. IOW you can compute an approximate unscaled coordinate of the best match in the source.

Visual interpretation of the results

Colors:

• A blacker result means the best match for the pixel came from the upper left of the selection in the source.
• A redder result means the best match for the pixel came from the upper right of the selection in the source.
• A greener result means the best match for the pixel came from the lower left of the selection in the source.
• A yellower result means the best match for the pixel came from the lower right of the selection in the source.

Uniformity:

• a smooth region indicates the region was synthesized from a continguous region of the source
• a pixelated region indicates the region was synthesized from many different places in the source

There is more to say here, but for now, I have limited experience with this feature.

(TODO are the coordinates in the frame of the selection in the source image, or the entire source image??)

The coordinates are in the frame of the source image. One can interpret coordinates as vectors from the origin (upper left) of the source image. If the target and source image are the same, then one could compute an offset vector (from a pixel in the direction of its best match.) But that is not what this feature does, it would need to be a post-processing step only for the given case that the target and source image are the same.

Non-determinism

The Resynthesizer uses a PRNG (pseudo random number generator.) Also, the Resynthesizer built threaded is non-deterministic (won't return the same result for consecutive runs with the same inputs.) And since you must run Resynthesizer twice to see both the usual result and the new result, for now it is difficult or impossible to get a new result that actually represents the exact same result as the usual result. Loosely speaking, they represent similar "resynthesizings" for a given set of inputs (images and parameters.)