Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-time Video Prediction With Fast Video Interpolation Model and Prediction Training

Shota Hirose, Kazuki Kotoyori, Kasidis Arunruangsirilert, Fangzheng Lin, Heming Sun, Jiro Katto

TL;DR

This work tackles the challenge of achieving real-time, zero-latency video transmission by predicting future frames. It introduces IFRVP, an efficient extension of IFRNet, and three training strategies—Recurrent, Arbitrary, and Independent Prediction—to adapt frame interpolation models for prediction, plus an ELAN-based residual block to reduce computation. Across Cityscapes experiments, IFRVP achieves state-of-the-art MS-SSIM with favorable speed-accuracy trade-offs, and IFRVP-Fast further reduces FLOPs to under 10 GFLOPs, enabling edge deployment. The results demonstrate practical viability for near-zero-latency interactive transmission and provide a path toward scalable, hardware-friendly video prediction for real-time networks.

Abstract

Transmission latency significantly affects users' quality of experience in real-time interaction and actuation. As latency is principally inevitable, video prediction can be utilized to mitigate the latency and ultimately enable zero-latency transmission. However, most of the existing video prediction methods are computationally expensive and impractical for real-time applications. In this work, we therefore propose real-time video prediction towards the zero-latency interaction over networks, called IFRVP (Intermediate Feature Refinement Video Prediction). Firstly, we propose three training methods for video prediction that extend frame interpolation models, where we utilize a simple convolution-only frame interpolation network based on IFRNet. Secondly, we introduce ELAN-based residual blocks into the prediction models to improve both inference speed and accuracy. Our evaluations show that our proposed models perform efficiently and achieve the best trade-off between prediction accuracy and computational speed among the existing video prediction methods. A demonstration movie is also provided at http://bit.ly/IFRVPDemo. The code will be released at https://github.com/FykAikawa/IFRVP.

Real-time Video Prediction With Fast Video Interpolation Model and Prediction Training

TL;DR

This work tackles the challenge of achieving real-time, zero-latency video transmission by predicting future frames. It introduces IFRVP, an efficient extension of IFRNet, and three training strategies—Recurrent, Arbitrary, and Independent Prediction—to adapt frame interpolation models for prediction, plus an ELAN-based residual block to reduce computation. Across Cityscapes experiments, IFRVP achieves state-of-the-art MS-SSIM with favorable speed-accuracy trade-offs, and IFRVP-Fast further reduces FLOPs to under 10 GFLOPs, enabling edge deployment. The results demonstrate practical viability for near-zero-latency interactive transmission and provide a path toward scalable, hardware-friendly video prediction for real-time networks.

Abstract

Transmission latency significantly affects users' quality of experience in real-time interaction and actuation. As latency is principally inevitable, video prediction can be utilized to mitigate the latency and ultimately enable zero-latency transmission. However, most of the existing video prediction methods are computationally expensive and impractical for real-time applications. In this work, we therefore propose real-time video prediction towards the zero-latency interaction over networks, called IFRVP (Intermediate Feature Refinement Video Prediction). Firstly, we propose three training methods for video prediction that extend frame interpolation models, where we utilize a simple convolution-only frame interpolation network based on IFRNet. Secondly, we introduce ELAN-based residual blocks into the prediction models to improve both inference speed and accuracy. Our evaluations show that our proposed models perform efficiently and achieve the best trade-off between prediction accuracy and computational speed among the existing video prediction methods. A demonstration movie is also provided at http://bit.ly/IFRVPDemo. The code will be released at https://github.com/FykAikawa/IFRVP.

Paper Structure

This paper contains 20 sections, 8 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Video Prediction
  4. Frame Interpolation and Extrapolation
  5. Zero-Latency Interaction by Video Prediction
  6. Efficient Layer Aggregation Network (ELAN)
  7. Proposed Method
  8. Training Extrapolation Models for Video Prediction
  9. Recurrent Prediction Training
  10. Arbitrary Prediction Training
  11. Independent Prediction Training
  12. Comparison of Training Models
  13. ELAN-based Residual Block for IFRVP
  14. EXPERIMENTS
  15. Experiment Setup
  16. ...and 5 more sections

Figures (6)

  • Figure 1: MS-SSIM on Cityscapes cityscapes (resized to 512$\times$1024) vs. computational complexity of various video prediction models. Proposed methods are marked as $\mathbin{\vcenter{\hbox{$\m@th\bullet$}}}$. We use the reported values from DMVFN for other methods.
  • Figure 2: Latency compensation by video prediction.
  • Figure 3: Illustrations of the three proposed training methods.
  • Figure 4: Architecture comparisons of various residual blocks. Estimated FLOPs is written in convolution block
  • Figure 5: Qualitative prediction comparison on Cityscapes cityscapes.
  • ...and 1 more figures