Appendix OBMC - AlexBoswellVCD/gitlab_wiki_test GitHub Wiki

Overlapped Block Motion Compensation Appendix

1. Description of the algorithm

For a given predicted block, where prediction is generated based on motion estimation MVs, the general idea behind OBMC is to consider additional predictors based on neighboring block MVs, as such predictors may offer more accurate predictions for pixels near the block boundary. A final inter predictor is generated using a weighted sum of current block predictor and predictors based on neighboring block MV. The weights favor the predictors based on the neighboring block near the block boundary and favor the current predictor for the pixels away from the boundary.

In general, OBMC produces better and smoother predictions near the boundary and may help reduce blocking artifacts. It may help also in the selection of larger inter partitions. OBMC is enabled only for inter blocks with single MV and with block sizes ≥ 8x8.

The prediction algorithm for a given block proceeds as follows:

  • Generate the motion compensated prediction Prediction_0 for the current block.

  • Consider the MVs from the N neighboring top and left inter coded blocks. For each of the N neighboring inter coded blocks B_i, i=1…N,

  • Define an overlap region, which is a region in the current block that extends along the boundary of the block B_i in one direction and extends half the current block width or height in the other direction.

  • Generate a prediction Prediction_i for the overlap area using the MVs for the block B_i.

  • Each pixel in the overlap area is computed as the weighted sum of the pixel in Prediction_0 and the pixel in Prediction_i, where the weights are generated using a raised cosine window function.

As an example, consider the yellow block shown in Figure 1 below. The 16x16 yellow block is the current block to be predicted. The 16x16 green and 8x8 blue blocks are inter coded neighboring blocks. The initial inter prediction for the yellow block is generated using regular inter prediction.

Figure 1. The yellow block in the figure is the inter block of interest.

The algorithm then proceeds through two main steps: Vertical blending and horizontal blending.

Step 1: Vertical blending (see Figure 2 below)

  • Consider now the area below the blue block and that extends half-way into the yellow block (The area inside the dotted blue line).

  • Generate inter prediction for the blue dotted block using the inter mode for the blue block.

  • Blend the inter predictions based on the blue block inter mode with that of the yellow block inter mode.

  • The blending is performed using the mask {36, 42, 48, 53, 57, 61, 64, 64}, where the resulting sample value is given by:

    [(64-mask(i))*blue_prediction + mask(i)*yellow_prediction]/64,

    where the index i corresponds to the y position of the pixel in the block.

Figure 2. Vertical blending of the prediction based on the blue 8x8 inter block.

Step 2: Horizontal blending (See Figure 3 below)

  • Consider now the area to the left of the light green block and that extends half-way into the yellow block (The area inside the dotted green line). Note that the orange area corresponds to the pixels that have already been updated in the vertical blending in step 1.

  • Generate inter prediction the dotted green block using the inter mode for the green block.

  • Blend the inter prediction based on the green block inter mode with that of the yellow/orange block inter mode.

  • The blending is performed using the mask {36, 42, 48, 53, 57, 61, 64, 64} and the final predicted pixel value is given by (64-mask(i))*blue_prediction + mask(i)*yellow_prediction or (64-mask(i))*blue_prediction + mask(i)*orange_prediction

where the index i corresponds to the x position of the pixel in the block.

Figure 3. Horizontal blending of the prediction based on the green inter block.

Notes:

  • If the prediction mode for one of the neighboring blocks is inter-inter compound then only the first MV is considered to build the neighboring overlapping prediction.

  • If the prediction mode for one of the neighboring blocks is inter-intra compound then only inter prediction is considered to build the neighboring overlapping prediction.

  • If the prediction mode for one of the neighboring blocks is warped then simple motion is considered to build the neighboring overlapping prediction.

  • OBMC requires the interpolation filter information to be stored as part of neighbor information.

2. Implementation of the algorithm

Control macros/flags:

Flags associated with the global motion compensation are listed in Table 1.

Table 1. Control flags for global motion compensation.
Flag Level (sequence/Picture) Description
enable_obmc Sequence Config flag to enable/disable OBMC at the sequence level
pic_obmc_mode Picture Picture based obmc mode

Candidates Generation:

OBMC is invoked from in the SB loop in the enc_dec_kernel through the functions mode_decision_sb and av1_encode_pass, as indicated in Figure 4 below. A more detailed diagram of the function calls in MD and in the encoder pass is given in Figure 5.

Figure 4. Main function calls from within the enc_dec_kernel where the function calls behind the generation and coding of OBMC candidates are highlighted.
Figure 5. More detailed diagram of the function calls leading to the generation and coding of OBMC candidates both in MD and in the encode pass.

In mode decision, two functions are used to generate OBMC candidates, namely inject_inter_candidates and av1_inter_prediction. The first is used in the generation of OBMC input candidates to be injected in MD stage 0. The second is used to generate OBMC predictions for MD stages 0/1/2. A special class is assigned to all OBMC candidates.

The steps involved in the generation of OBMC input candidates for MD stage 0 are as follows:

  1. Determine the number of top blocks and the number of blocks to the left of the current block that overlap with the current block (eb_av1_count_overlappable_neighbors). A top block is considered to overlap with the current block if it is an inter block. Similar idea applies to blocks to the left of the current block.

  2. Check if the block satisfies the following conditions so that it would be considered in the generation of corresponding OBMC MD candidates (obmc_motion_mode_allowed):

  • Block height and block width are both greater than or equal to 8.

  • The inter mode of the block is single-reference mode (i.e. single reference NEW/NEAR/ NEAREST).

  • There are overlapping blocks on top or to the left of the current block.

If the above conditions are satisfied, an MD candidate is created with motion_mode field set to OBMC_CAUSAL.

  1. Generate predictions for the current block based on the top and left blocks (precompute_obmc_data). The generation of these predictions is performed as follows:

  • For each of the overlapping top blocks, prediction candidate for the current block based on the MV of the top block is generated. (build_prediction_by_above_preds)

  • For each of the overlapping top blocks, prediction candidate for the current block based on the MV of the top block is generated. (build_prediction_by_left_preds)

The predictions for the current block based on the MVs of the top blocks and those of the left blocks are prepared in the above steps to be used in subsequent stages when the actual OBMC prediction is computed.

The generation of OBMC predictions using the function av1_inter_prediction in MD stages 0, 1 and 2 proceeds along the same lines as in precompute_obmc_data to compute predictions for the current block based on MVs of the top and left blocks. Blending of those predictions is then performed to construct the OBMC prediction.

Similarly, in the encode_pass, the function av1_inter_prediction is called to generate the final OBMC encodings based on the final partitioning and modes for the neighboring blocks.

4. Optimization of the algorithm

The optimization of the OBMC algorithm is based on considering the tradeoff between the number of OBMC candidates that are allowed into MD stage 2 and the quality gains provided by the feature. The tradeoffs are controlled by the pic_obmc_mode flag. Table 1 below provides a description of the quality-complexity tradeoffs in the algorithm.

Table 2. Description of the settings of the pic_obmc_mode flag.
pic_obmc_mode Description
0 OFF
1 ON best quality: OBMC @(MVP, PME and ME) + 16 NICs
2 ON Fast: OBMC @(MVP, PME and ME) + Optimized NICs
3 ON Faster: OBMC @(MVP, PME ) + Optimized NICs
4 ON Fastest: OBMC @(MVP, PME ) + more Optimized NICs

Note: NIC refers to the number of input OBMC candidates into MD stage 2 (i.e. the final MD stage)

For the case where (sc_content_detected == 0 && slice_type != I_SLICE), pic_obmc_mode is given in Table 2 below for the different encoder presets, otherwise pic_obmc_mode = 0.

Table 3. pic_obmc_mode as a function of encoder preset.
Encoder Preset (enc_mode) pic_obmc_mode
0 2
1 to 8 0

5. Signaling

is_motion_mode_switchable: specifying that motion mode can change from one block to another. As a result a block based field called motion_mode is sent to indicate the type of the motion mode: SIMPLE_TRANSLATION / OBMC / WARPED.

References

Yue Chen and Debargha Mukherjee, “Variable block-size overlapped block motion compensation in the next generation open-source video codec,” International Conference on Image Processing, pp. 938-942, 2017