videofilt_prefetchingmultiplesourceframes - shekh/VirtualDub2 GitHub Wiki

VirtualDub Plugin SDK 1.2

Prefetching multiple source frames

By default, VirtualDub video filters operate in 1:1 mode, where each source frame corresponds to exactly one output frame. The output frame stream has the same frame count, frame rate, and frame timings as the source stream.

V13 or earlier: Prefetching multiple source frames per output frame is only possible starting with API V14. If you are targeting an earlier API version, you need to use alternative ways of handling this, such as using internal buffering. See setting filter parameters on how to declare an internal lagging delay in your filter to accommodate this.

Requesting multiple source frames

In order to receive multiple source frames, a filter must implement the prefetchProc2 method. This method receives a prefetcher object which is invoked by the filter for each source frame required.

A filter does not need to check for source frame numbers lying outside of the source stream when prefetching. Any frame numbers that are outside of the stream are clamped to the beginning or end as appropriate. That means, for instance, that a filter that needs a five-frame window can simply request frames -2 and -1 at the start of the window and receive frame 0 twice in those slots. The frames will be identified as frame 0, however.

The prefetch pattern can be different for each output frame, and a filter is not required to prefetch the same number of source frames each time. The filter does have to prefetch any frame that it might need in the runProc() method, though, because by the time that runs it is too late to fetch any more frames.

VirtualDub specific: Starting with VirtualDub 1.9.1, the filter system caches frames within the filter pipeline. This minimizes the cost of redundantly fetching source frames.

Processing with multiple source frames

Normally, the runProc method receives input and output frames via the src and dst fields of the VDXFilterActivation structure. The V14 API adds the mpInputFrames and mpOutputFrames arrays, which contain pointers to source and output frames. The src and dst fields alias onto the first entry of those arrays, whereas any additional frames take up additional slots.

If your filter does not need more than one source frame, it can continue to use src and dst.

Identifying source frames

Source frames always arrive in the same order that they are prefetched. If you are making a filter that uses a window of source frames, you can request them in order and rely on that order being consistent. The host can choose to read frames out of order, but this is invisible to your filter.

In cases where position is not enough, such as when some output frames require more source frames than others, you can supply a cookie to the PrefetchFrame() method. The cookie is an arbitrary 64-bit integer that is copied into the mCookie field of the VDXFBitmap structure of the corresponding frame when runProc() is called. One way to use this is to store an index into the cookie and use that to sort the frames into an array in the run method.

Source frames are also tagged by their frame number in the mFrameNumber field. Note, however, that this is the true frame number, which is different from the requested frame number when the frame number has had to be clamped to be within the stream. In other words, a request for frame -2 will return a frame that is tagged as frame 0.

Fetching frames in direct mode

With the prefetcher object, you can also invoke the PrefetchFrameDirect() method to indicate that a frame should be copied directly from source to output. In addition to speeding up rendering for frames that your filter does not need to process, this also permits smart rendering logic to bypass the entire filter chain whenever possible and copy compressed frames directly from source. This runs much faster and preserves full quality in the source. If your filter is designed to only process a portion of its video, you should use PrefetchFrameDirect() whenever possible to enable direct mode operation.

Note that even if you mark a frame as direct, your filter's runProc method may still be called. One reason this may occur is if the frame cannot be copied in direct mode. When this happens, your filter should copy the frame as required.

Direct mode requires that source and output frame formats be compatible. If the formats or frame sizes do not match, it is a logic error to request a direct copy. You can, however, request a direct copy between streams that are of different frame rates or aspect ratios.

Prefetching symbolic frames

A tricky filter can use cached information to determine that some source frames may not be necessary. This is particularly possible if the filter has already processed an output frame before. For instance, a filter may have cached the location of a detected scene transition, and avoid prefetching source frames across that transition. Although caching of source frames helps, avoiding prefetches entirely reduces CPU usage in the filter system.

The problem with doing this is that it can obscure the relationship between frames in the filter pipeline and sometimes the host needs to know that frame 100 after the filter comes from frame 50 in the source, perhaps for audio purposes. The filter can do a symbolic prefetch via the PrefetchFrameSymbolic() to declare the equivalent source frame without actually doing a frame fetch. This then allows the host to cheaply follow frame dependencies.

Symbolic fetches are optional. If the filter does not do a symbolic fetch, the host assumes that the symbolic frame is the mean of the prefetched source frames, or non-existent if no source frames are requested. A direct frame automatically becomes the symbolic frame. If the filter both issues regular frame prefetches and a symbolic prefetch, the symbolic prefetch takes precedence. Therefore, it is safe simply to always do the symbolic prefetch.

Prefetch in capture mode (VirtualDub specific)

The prefetcher allows a filter to request source frames in entirely arbitrary order, but this can cause problems when the filter chain must be run in push mode. This happens if filters are enabled during live video capture. In this case, VirtualDub attempts to maintain a queue of source frames to satisfy prefetch requests. If a frame is prefetched that is too old and has already scrolled off the queue, the oldest available frame is supplied instead. Whenever possible, filters should attempt to be monotonic in their prefetches to support this. More specifically, it is best that the oldest source frame required never go backward when prefetching successive output frames.