Object Segmentation-Assisted Inter Prediction for Versatile Video Coding

TL;DR

This work proposes an object segmentation-assisted inter prediction method (SAIP), where objects in the reference frames are segmented by some advanced technologies to address the problem of multiple moving objects of arbitrary shapes in natural videos.

Abstract

In modern video coding standards, block-based inter prediction is widely adopted, which brings high compression efficiency. However, in natural videos, there are usually multiple moving objects of arbitrary shapes, resulting in complex motion fields that are difficult to represent compactly. This problem has been tackled by more flexible block partitioning methods in the Versatile Video Coding (VVC) standard, but the more flexible partitions require more overhead bits to signal and still cannot be made arbitrarily shaped. To address this limitation, we propose an object segmentation-assisted inter prediction method (SAIP), where objects in the reference frames are segmented by some advanced technologies. With a proper indication, the object segmentation mask is translated from the reference frame to the current frame as the arbitrary-shaped partition of different regions without any extra signal. Using the segmentation mask, motion compensation is separately performed for different regions, achieving higher prediction accuracy. The segmentation mask is further used to code the motion vectors of different regions more efficiently. Moreover, the segmentation mask is considered in the joint rate-distortion optimization for motion estimation and partition estimation to derive the motion vector of different regions and partition more accurately. The proposed method is implemented into the VVC reference software, VTM version 12.0. Experimental results show that the proposed method achieves up to 1.98%, 1.14%, 0.79%, and on average 0.82%, 0.49%, 0.37% BD-rate reduction for common test sequences, under the Low-delay P, Low-delay B, and Random Access configurations, respectively.

Figures (9)

• Figure 1: The partition results of different methods in the actual coding process, where (a) uses the rectangular partition (RP), (b) uses the RP + line-based geometric partition (GP), (c) uses the RP + segmentation-based partition (SP). The block is in the 71-th frame of the BQMall sequence.
• Figure 2: Illustration of the proposed object segmentation-assisted inter prediction (SAIP) framework from the perspective of decoding. Boxes represent the sub-modules of SAIP, and arrows indicate the data flow direction. Section indicates the corresponding chapter of the module.
• Figure 3: The object segmentation workflow on the reference frames.
• Figure 4: Schematic representation of the segmentation-assisted motion compensation, including integer and fractional parts.
• Figure 5: An example of overlapped region-based motion compensation. The red line is the partition line of motion-inconsistent regions.