OpenCV-Python for VapourSynth

This tutorial focuses on the installation and application of OpenCV-Python for VapourSynth.

Installation with pip

The installation part targets Windows platform, but similar methods should also work on other platforms.

1. Install pip (It should have been installed after you install Python normaly) and make sure it works (for example, type pip -V in the shell to check the version of pip).

2. Compile NumPy and OpenCV-Python, or download prebuilt packages from NumPy and OpenCV-Python. Or simply run pip install numpy opencv-python (or pip install numpy opencv-python opencv-contrib-python if you need the contrib module) and then jump to 4.

3. Install the two packages by typing pip install *.whl in the shell.

4. Check installation by typing

import numpy as np
print(np.__version__)

import cv2
print(cv2.__version__)

in Python terminal.

Using OpenCV-Python in VapourSynth

The functions in OpenCV-Python cannot be involked by simply typing clip = cv2.GaussianBlur(clip) since they work on NumPy arrays. To use them conveniently, I have written the muvsfunc_numpy.py to support the usage.

To make OpenCV-Python's functions work on VapourSynth, you only need to define a function (pipeline) which inputs one or many NumPy arrays and output one NumPy array, as demonstrated in the following examples.

It should be noted that different OpenCV-Python's filters have different requirments on the formats of inputs. You'd better check the official documentation of OpenCV before you use them.

In the following examples, I assume that libraries have been imported by

import numpy as np
import cv2
import muvsfunc_numpy as mufnp

Gaussian blur on grayscale

Pure VapourSynth:

sigma = 1.0
output = core.tcanny.TCanny(gray, sigma=sigma, mode=-1)

OpenCV-Python:

sigma = 1.0
ksize = (round(sigma*3)*2+1, round(sigma*3)*2+1)
gaussian_core = lambda img: cv2.GaussianBlur(img, ksize=ksize, sigmaX=sigma)
output = mufnp.numpy_process(gray, gaussian_core)

I like to denote a function which inputs and outputs NumPy data by suffix _core.

Detail enhancement on RGB using Domain Transform

To the best of my knowledge, there is no implementation of DT on VapourSynth. Thus OpenCV-Python with a large library of filters greatly increases the choices of filters.

OpenCV-Python:

# enhance_core = lambda img: cv2.detailEnhance(img)
enhace_core = cv2.detailEnhance # aliasing
output = mufnp.numpy_process(rgb, enhance_core, input_per_plane=False, output_per_plane=False)
# ***_per_plane=False is required since we filter all of the three channels simultaneously

Canny edge detector

Filters that all belong to OpenCV-Python library can be placed in a single function. This enables complex pipeline and also might improves the performance.

Pure VapourSynth:

sigma = 1.0
output = core.tcanny.TCanny(gray, sigma=sigma, mode=0)

OpenCV-Python:

# This time we define a function explicitly rather than using anonymous function
def canny_core(img):
    blur = cv2.GaussianBlur(img, ksize=(7, 7), sigmaX=1)
    edge = cv2.Canny(blur, threshold1=1.0, threshold2=8.0)
    return edge

output = mufnp.numpy_process(gray, canny_core)

(The two results different from each other. I don't figure out the reason and also not care about it.)

Resize

Pure VapourSynth:

output = core.resize.Bicubic(gray, 1280, 720)

Resize or other operator in OpenCV-Python which alter the format of the input require a special treatment on the input of mufnp.numpy_process as the following, since the format of the first input and the output of VapourSynth's std.ModifyFrame should be identical.

OpenCV-Python (v1):

def resize_core1(blank, img):
    return cv2.resize(img, dsize=(blank.shape[1], blank.shape[0]), interpolation=cv2.INTER_CUBIC)

blank_clip = core.std.BlankClip(gray, width=1280, height=720)
output = mufnp.numpy_process([blank_clip, gray], resize_core1)

OpenCV-Python (v2):

def resize_core2(img, w, h):
    return cv2.resize(img, dsize=(w, h), interpolation=cv2.INTER_CUBIC)

blank_clip = core.std.BlankClip(gray, width=1280, height=720)
output = mufnp.numpy_process([blank_clip, gray], resize_core2, w=1280, h=720, omit_first_clip=True)

(Again, the results are not identical.)

Pencil Drawing Production

OpenCV-Python (grayscale output):

pencil_core1 = lambda img: cv2.pencilSketch(img)[0]
blank_clip = core.std.BlankClip(rgb, format=vs.GRAY8)
output = mufnp.numpy_process([blank_clip, rgb], pencil_core1, input_per_plane=False, omit_first_clip=True)

OpenCV-Python (color output):

pencil_core2 = lambda img: cv2.pencilSketch(img)[1]
output = mufnp.numpy_process(rgb, pencil_core2, input_per_plane=False, output_per_plane=False)

Guided Filter

OpenCV-Python (use grayscale to guide the filtering on RGB):

gf_core = lambda img, guide: cv2.ximgproc.guidedFilter(guide, img, radius=4, eps=100)
output = mufnp.numpy_process([rgb, gray], gf_core, input_per_plane=False, output_per_plane=False)

OpenCV-Python (use RGB to guide the filtering on RGB):

gf_core = lambda img, guide: cv2.ximgproc.guidedFilter(guide, img, radius=4, eps=100)
output = mufnp.numpy_process([rgb, rgb], gf_core, input_per_plane=False, output_per_plane=False)

Inpainting

OpenCV-Python:

def inpaint_core(img, mask, radius=1, flags=cv2.INPAINT_NS):
    return cv2.inpaint(img, mask, inpaintRadius=radius, flags=flags)

inpainting = mufnp.numpy_process([gray, mask], inpaint_core, radius=1, flags=cv2.INPAINT_NS)
inpainting = core.std.MaskedMerge(gray, inpainting, mask)

Limitation

1. Currently the temporal filters do not work. Specifically, I do not figure out how to use them inside VapourSynth's std.ModifyFrame, which is the filter used to support NumPy-based filters. One solution is to input multiple time-shifted clips.