Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advanced Learning-Based Inter Prediction for Future Video Coding

Yanchen Zhao, Wenhong Duan, Chuanmin Jia, Shanshe Wang, Siwei Ma

TL;DR

The paper proposes a low complexity learning-based inter prediction (LLIP) method to replace the traditional INTERPF, which enhances the filtering process by leveraging a lightweight neural network model, where parameters can be exported for efficient inference.

Abstract

In the fourth generation Audio Video coding Standard (AVS4), the Inter Prediction Filter (INTERPF) reduces discontinuities between prediction and adjacent reconstructed pixels in inter prediction. The paper proposes a low complexity learning-based inter prediction (LLIP) method to replace the traditional INTERPF. LLIP enhances the filtering process by leveraging a lightweight neural network model, where parameters can be exported for efficient inference. Specifically, we extract pixels and coordinates utilized by the traditional INTERPF to form the training dataset. Subsequently, we export the weights and biases of the trained neural network model and implement the inference process without any third-party dependency, enabling seamless integration into video codec without relying on Libtorch, thus achieving faster inference speed. Ultimately, we replace the traditional handcraft filtering parameters in INTERPF with the learned optimal filtering parameters. This practical solution makes the combination of deep learning encoding tools with traditional video encoding schemes more efficient. Experimental results show that our approach achieves 0.01%, 0.31%, and 0.25% coding gain for the Y, U, and V components under the random access (RA) configuration on average.

Advanced Learning-Based Inter Prediction for Future Video Coding

TL;DR

The paper proposes a low complexity learning-based inter prediction (LLIP) method to replace the traditional INTERPF, which enhances the filtering process by leveraging a lightweight neural network model, where parameters can be exported for efficient inference.

Abstract

In the fourth generation Audio Video coding Standard (AVS4), the Inter Prediction Filter (INTERPF) reduces discontinuities between prediction and adjacent reconstructed pixels in inter prediction. The paper proposes a low complexity learning-based inter prediction (LLIP) method to replace the traditional INTERPF. LLIP enhances the filtering process by leveraging a lightweight neural network model, where parameters can be exported for efficient inference. Specifically, we extract pixels and coordinates utilized by the traditional INTERPF to form the training dataset. Subsequently, we export the weights and biases of the trained neural network model and implement the inference process without any third-party dependency, enabling seamless integration into video codec without relying on Libtorch, thus achieving faster inference speed. Ultimately, we replace the traditional handcraft filtering parameters in INTERPF with the learned optimal filtering parameters. This practical solution makes the combination of deep learning encoding tools with traditional video encoding schemes more efficient. Experimental results show that our approach achieves 0.01%, 0.31%, and 0.25% coding gain for the Y, U, and V components under the random access (RA) configuration on average.

Paper Structure

This paper contains 11 sections, 8 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Inter Prediction Filter in AVS
  3. Proposed Method
  4. Network Structure Analysis
  5. Model Inference Library
  6. Implementation in EVM
  7. Experimental Results
  8. Traning Process
  9. Coding Gain Analysis
  10. Coding Complexity Analysis
  11. Conclusion

Figures (2)

  • Figure 1: Inter Prediction Filter (INTERPF) in Audio Video coding Standard (AVS) traverses each pixel $P$ in the current Coding Unit (CU) block and then filters pixel $P$ using its coordinate values and adjacent four reconstructed pixel values.
  • Figure 2: Illustration of the low complexity learning-based inter prediction network architecture. The figure shows two different schemes, with "Scheme 1" on the left, which only inputs the values of the adjacent reconstructed pixels and the current pixel. On the right is "Scheme 2", which adds the coordinates of the current pixel in addition to "Scheme 1".