Motion-Plane-Adaptive Inter Prediction in 360-Degree Video Coding

TL;DR

A motion-plane-adaptive inter prediction technique (MPA) for 360-degree video that takes the spherical characteristics of 360-degree video into account and derives a motion-plane-adaptive motion vector prediction technique (MPA-MVP) that allows to translate motion information between different motion planes and motion models.

Abstract

Inter prediction is one of the key technologies enabling the high compression efficiency of modern video coding standards. 360-degree video needs to be mapped to the 2D image plane prior to coding in order to allow compression using existing video coding standards. The distortions that inevitably occur when mapping spherical data onto the 2D image plane, however, impair the performance of classical inter prediction techniques. In this paper, we propose a motion-plane-adaptive inter prediction technique (MPA) for 360-degree video that takes the spherical characteristics of 360-degree video into account. Based on the known projection format of the video, MPA allows to perform inter prediction on different motion planes in 3D space instead of having to work on the - in theory arbitrarily mapped - 2D image representation directly. We furthermore derive a motion-plane-adaptive motion vector prediction technique (MPA-MVP) that allows to translate motion information between different motion planes and motion models. Our proposed integration of MPA together with MPA-MVP into the state-of-the-art H.266/VVC video coding standard shows significant Bjontegaard Delta rate savings of 1.72% with a peak of 3.97% based on PSNR and 1.56% with a peak of 3.40% based on WS-PSNR compared to the VTM-14.2 baseline on average.

Figures (12)

• Figure 1: (a) A 360-degree image mapped to the 2D image plane using an equirectangular projection (ERP). (b), (c) The 360-degree image mapped to the unit sphere in 3D space visualizing (b) the front view and (c) the back view.
• Figure 2: The employed 3D coordinate systems. The black coordinate system $(x, y, z)$ describes the main system orientation where $y$ is oriented horizontally, $z$ is oriented vertically, and $x$ is oriented perpendicular to $y$ and $z$. The default camera is positioned at the origin and oriented in negative $x$-direction. $\theta$ and $\varphi$ denote the corresponding polar and angular angles in spherical coordinates. The blue coordinate system $(x^\prime, y^\prime, z^\prime)$ describes an intermediate system used for the perspective projection where the virtual perspective camera is oriented in positive $z^\prime$-direction. The corresponding polar and angular angles $\theta^\prime$ and $\varphi^\prime$ are given in blue.
• Figure 3: Perspective image planes. Light rays with incident angles $\theta^\prime < \pi/2$ are projected to the real image plane, while light rays with incident angles $\theta^\prime > \pi/2$ are projected to the virtual image plane.
• Figure 4: Schematic representation of the motion-plane-adaptive motion model with a visualization of the procedure for an exemplary block in a 360-degree image employing an ERP projection. For visualization, block motion for an exemplary motion vector and rotation matrix is shown. The original block is shown in orange in the top row and the corresponding moved block is shown in blue in the bottom row. The applied motion plane rotation matrix rotates the motion plane by $\pi/2$ around the $y$-axis. It is clearly visible that the distortions in the equirectangular domain resulting from translational motion on the street surface are accurately replicated by the motion-plane-adaptive motion model.
• Figure 5: Spatial motion vector predictor candidates in the H.266/VVC video coding standard VVC-Draft10 with a schmematic illustration of MPA-MVP for candidate position B1 at pixel coordinate $\boldsymbol{p}_\text{s}$. With $\boldsymbol{R}_\text{s}$ describing the motion plane of the candidate position and $\boldsymbol{R}_\text{t}$ describing the motion plane of the current block, MPA-MVP translates the motion vector $\boldsymbol{t}_\text{s}$ from B1 to the corresponding motion vector predictor $\boldsymbol{t}_\text{t}$ for the current block.