RIFE - AmusementClub/vs-mlrt GitHub Wiki

RIFE is a Real-time Intermediate Flow Estimation algorithm for video frame interpolation. It supports arbitrary-timestep interpolation between a pair of inputs.

Link:

Includes these models:

  • v4.0, v4.2, v4.3, v4.4, v4.5, v4.6, v4.7, v4.8, v4.9, v4.10: RGB frame interpolation

Requirements & Parameters

  1. block_w and block_h (tile size) must be multiples of 32 / scale.

vsmlrt.py wrapper Usage

In order to simplify usage, we provided a Python wrapper module vsmlrt:

from vsmlrt import RIFE, RIFEModel, Backend

src = core.std.BlankClip(format=vs.RGBS)

# backend could be:
#  - CPU Backend.OV_CPU(): the recommended CPU backend; generally faster than ORT-CPU.
#  - CPU Backend.ORT_CPU(num_streams=1, verbosity=2): vs-ort cpu backend.
#  - GPU Backend.ORT_CUDA(device_id=0, cudnn_benchmark=True, num_streams=1, verbosity=2)
#     - use device_id to select device
#     - set cudnn_benchmark=False to reduce script reload latency when debugging, but with slight throughput performance penalty.
#  - GPU Backend.TRT(fp16=True, device_id=0, num_streams=1): TensorRT runtime, the fastest NV GPU runtime.
flt = RIFE(src, model=RIFEModel.v4_4, backend=Backend.ORT_CUDA())

Raw Model Usage

not recommended

Benchmarking

Measurements: FPS / Device Memory (MB)

Device memory:

  • GPU: device memory including context

RTX 3090

Software: VapourSynth R60, Windows Server 2019, Graphics Driver 526.47.

Input size: 1920x1088

Backends

  1. vs-mlrt v12

Performance

FP16

Model [1] ort-cuda [1] trt [1] ort-cuda (2 streams) [1] trt (2 streams)
v4.4 53.62 / 1771 71.30 / 626 83.34 / 2926 107.3 / 962