Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AV1 Motion Vector Fidelity and Application for Efficient Optical Flow

Julien Zouein, Vibhoothi Vibhoothi, Anil Kokaram

TL;DR

This work investigates using AV1-encoded motion vectors as a computationally efficient substitute for optical flow in computer vision tasks. It systematically evaluates MV fidelity against HEVC and ground-truth flow, and demonstrates how these vectors can serve as a warm-start to refine with a deep optical flow model (RAFT), achieving substantial speedups with minimal accuracy loss. The study highlights encoder preset effects on motion fidelity, showing that certain AV1 configurations offer competitive quality and high perceptual metrics like VMAF. Practically, reusing bitstream motion vectors can accelerate motion-aware applications such as frame interpolation and action recognition, enabling real-time or near-real-time pipelines when combined with RAFT refinement.

Abstract

This paper presents a comprehensive analysis of motion vectors extracted from AV1-encoded video streams and their application in accelerating optical flow estimation. We demonstrate that motion vectors from AV1 video codec can serve as a high-quality and computationally efficient substitute for traditional optical flow, a critical but often resource-intensive component in many computer vision pipelines. Our primary contributions are twofold. First, we provide a detailed comparison of motion vectors from both AV1 and HEVC against ground-truth optical flow, establishing their fidelity. In particular we show the impact of encoder settings on motion estimation fidelity and make recommendations about the optimal settings. Second, we show that using these extracted AV1 motion vectors as a "warm-start" for a state-of-the-art deep learning-based optical flow method, RAFT, significantly reduces the time to convergence while achieving comparable accuracy. Specifically, we observe a four-fold speedup in computation time with only a minor trade- off in end-point error. These findings underscore the potential of reusing motion vectors from compressed video as a practical and efficient method for a wide range of motion-aware computer vision applications.

AV1 Motion Vector Fidelity and Application for Efficient Optical Flow

TL;DR

This work investigates using AV1-encoded motion vectors as a computationally efficient substitute for optical flow in computer vision tasks. It systematically evaluates MV fidelity against HEVC and ground-truth flow, and demonstrates how these vectors can serve as a warm-start to refine with a deep optical flow model (RAFT), achieving substantial speedups with minimal accuracy loss. The study highlights encoder preset effects on motion fidelity, showing that certain AV1 configurations offer competitive quality and high perceptual metrics like VMAF. Practically, reusing bitstream motion vectors can accelerate motion-aware applications such as frame interpolation and action recognition, enabling real-time or near-real-time pipelines when combined with RAFT refinement.

Abstract

This paper presents a comprehensive analysis of motion vectors extracted from AV1-encoded video streams and their application in accelerating optical flow estimation. We demonstrate that motion vectors from AV1 video codec can serve as a high-quality and computationally efficient substitute for traditional optical flow, a critical but often resource-intensive component in many computer vision pipelines. Our primary contributions are twofold. First, we provide a detailed comparison of motion vectors from both AV1 and HEVC against ground-truth optical flow, establishing their fidelity. In particular we show the impact of encoder settings on motion estimation fidelity and make recommendations about the optimal settings. Second, we show that using these extracted AV1 motion vectors as a "warm-start" for a state-of-the-art deep learning-based optical flow method, RAFT, significantly reduces the time to convergence while achieving comparable accuracy. Specifically, we observe a four-fold speedup in computation time with only a minor trade- off in end-point error. These findings underscore the potential of reusing motion vectors from compressed video as a practical and efficient method for a wide range of motion-aware computer vision applications.
Paper Structure (12 sections, 1 equation, 4 figures, 1 table)

This paper contains 12 sections, 1 equation, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Extraction and pre-processing of the Motion Vectors
  4. Motion Vector Extraction and Optic Flow Initialisation
  5. Refinement of motion vectors
  6. Investigating the Impact of Encoder Speed Presets in AV1
  7. Experimental Setup
  8. Encoder parameters
  9. Experimental Results and Discussion
  10. Impact of encoder configuration
  11. Refinememt with RAFT
  12. Conclusion

Figures (4)

  • Figure 1: Visualisation of the processing of the extracted motion. Left to Right: RGB frame, Motion Field extracted from AV1, Processed Motion Field from AV1, Reference Optical Flow given by Spring Dataset.
  • Figure 2: Visualisation of the elements from Spring Dataset. Left to Right: Frame extracted from Sequence 38, High-detail map with white pixels corresponding to details, Rigid Map where Non-Rigid object is in white; Sky Map: where white pixels correspond to the sky.
  • Figure 3: Average End Point Error versus Average VMAF by CPU configuration.
  • Figure 4: Impact of using AV1 Motion Field as warm-start for Raft. Left: Global EPE of RAFT off the shelf; Right: Global EPE of Fine-tuned RAFT; Using AV1 vectors as warm start (red) does not affect final accuracy that much but always leads to better convergence regardless of RAFT refinement.