Artifacts - jcjohnson/neural-style GitHub Wiki

Artifacts Produced By Neural-Style:

This Wiki page is for describing the different kinds of artifacts that can be encountered when using Neural-Style (or potentially other style transfer projects), and solutions for dealing with these artifacts. Artifacts are generally not from style images.

Artifact Name: A descriptive name.

Description: Describe how the artifact can be identified.

Cause: What causes the artifact to occur.

Solution: How to prevent/mitigate the artifact and it's effects.

Examples: Example images showcasing the artifact.

Artifact Name: Gray Spots/Haze

Description: Gray haze which covers certain regions of the output image, or gray spots which can destroy parts of the output image as they grow.

Cause: Unstable parameters used with the Adam optimizer.

Solution: Using the information from here, setting the beta1 parameter to 0.99 and the epsilon parameter to 1e-1.

Examples:

Artifact Name: Gray Rectangles

Description: Gray rectangles form horizontally, or vertically, and sometimes with only one hard edge. They can be really short, or long.

Cause: Currently, I suspect that the FCN-32s PASCAL model's training causes these issues, and not any of Neural-Style's parameters.

Solution: Edit the output image in a third party image editor, or try to configure Neural-Style's parameters in a way that mitigates the gray rectangles.

Examples:

Artifact Name: Geometric Grid Patterns

Description: These can be identified as dots in a grid like pattern across the output image, or as geometric lines resembling the lower layers of some DeepDream models. These artifacts appear especially easy to see on NIN (Network In Network) models, and the ResNet models that I have tested.

Cause: The model architecture and/or size, in addition to training, are the likely causes.

Solution: Use a extremely low -tv_weight that's just large enough to obscure and thus remove the visible artifacts.

Artifact Name: Border Lines

Description: Long lines running parallel to the output image's borders. They are usually not continuous. Do not confuse them with the output images produced by style images with visible borders. They seem to appear most commonly at smaller image sizes.

Cause: I'm not entirely sure, but I think the padding may have something to do with it.

Solution: Using a Multiscale Resolution script can help render these artifacts almost invisible at larger image sizes.

Examples:

Artifact Name: Convolutional Square Artifacts

Description: A grid pattern of rectangular/square artifacts that cover the entire image. They are generally not visible without averaging a large amount of output images which used the same -image_size and -content_image values. Though they do appear to be extremely visible when using some real photographs as style images.

Cause: These artifacts are caused by how convolutional neural networks see images. Basically images are broken down into small pieces for the network to process, and these pieces are visualized as rectangular/square artifacts on the output image. This GIF shows a visualization of how this works: https://i.stack.imgur.com/f2RiP.gif

Solution: Using a Multiscale Resolution script can help obscure these artifacts in some cases, but in other cases, I have yet to find a working solution.

Examples: