Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging AV1 motion vectors for Fast and Dense Feature Matching

Julien Zouein, Hossein Javidnia, François Pitié, Anil Kokaram

TL;DR

This work tackles the high computational burden of dense feature matching for structure-from-motion (SfM) on video by repurposing AV1 motion vectors to produce dense, sub-pixel correspondences and multi-frame tracks. The proposed pipeline converts motion vectors into sub-pixel point correspondences, propagates tracks with a cosine-consistency filter to extend coverage, and evaluates pairwise geometry using $E$ or $H$ with Sampson error-based scoring. Empirical results on a 117-frame clip show the MV-based front end registers all frames and yields substantially more 3D points with competitive reprojection error, while consuming far less CPU than SIFT-based baselines; a small SfM demonstration confirms practical viability. The findings suggest compressed-domain features from video codecs can serve as scalable, resource-efficient front ends for SfM, with future work focusing on full SfM and BA scaling and optimization.

Abstract

We repurpose AV1 motion vectors to produce dense sub-pixel correspondences and short tracks filtered by cosine consistency. On short videos, this compressed-domain front end runs comparably to sequential SIFT while using far less CPU, and yields denser matches with competitive pairwise geometry. As a small SfM demo on a 117-frame clip, MV matches register all images and reconstruct 0.46-0.62M points at 0.51-0.53,px reprojection error; BA time grows with match density. These results show compressed-domain correspondences are a practical, resource-efficient front end with clear paths to scaling in full pipelines.

Leveraging AV1 motion vectors for Fast and Dense Feature Matching

TL;DR

This work tackles the high computational burden of dense feature matching for structure-from-motion (SfM) on video by repurposing AV1 motion vectors to produce dense, sub-pixel correspondences and multi-frame tracks. The proposed pipeline converts motion vectors into sub-pixel point correspondences, propagates tracks with a cosine-consistency filter to extend coverage, and evaluates pairwise geometry using or with Sampson error-based scoring. Empirical results on a 117-frame clip show the MV-based front end registers all frames and yields substantially more 3D points with competitive reprojection error, while consuming far less CPU than SIFT-based baselines; a small SfM demonstration confirms practical viability. The findings suggest compressed-domain features from video codecs can serve as scalable, resource-efficient front ends for SfM, with future work focusing on full SfM and BA scaling and optimization.

Abstract

We repurpose AV1 motion vectors to produce dense sub-pixel correspondences and short tracks filtered by cosine consistency. On short videos, this compressed-domain front end runs comparably to sequential SIFT while using far less CPU, and yields denser matches with competitive pairwise geometry. As a small SfM demo on a 117-frame clip, MV matches register all images and reconstruct 0.46-0.62M points at 0.51-0.53,px reprojection error; BA time grows with match density. These results show compressed-domain correspondences are a practical, resource-efficient front end with clear paths to scaling in full pipelines.
Paper Structure (14 sections, 2 equations, 4 figures, 3 tables)

This paper contains 14 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Converting Motion Vectors into Sub-pixel Point-to-Point Correspondences
  4. Track Propagation and Cosine Filtering
  5. Pairwise Geometry Estimation and Scoring
  6. Experiments
  7. Experimental Setup
  8. Dataset
  9. SfM
  10. Encoder Parameters
  11. Baseline and Settings
  12. Main Qualitative Results
  13. Discussion and Limitations
  14. Conclusion

Figures (4)

  • Figure 1: Visualisation of the impact of using tracks on the adjacency matrix. (a): adjacency before using tracks, (b): adjacency after using tracks and cosine filtering. For a given row $i$ and column $j$, the whiter a pixel is, the more matches there are.
  • Figure 2: Pipeline from AV1 bitstream to correspondences and tracks. We parse block/motion/reference maps, generate sub-pixel point-to-point matches, build and filter tracks and export correspondences. Downstream SfM/BA is not the focus; we include a small demo in §III-C.
  • Figure 3: 3D reconstruction of Sequence Paris Seq 1. using MV (NVENC).
  • Figure 4: Process CPU Utilization (%) for each method on (a) KITTI-00, (b) Paris-2, and (c) Gerrard Hall, and (d) the corresponding per-image match counts. Codec-based motion-vector pipelines (NVENC-AV1, AOM-AV1) produce higher match counts than classical feature pipelines under identical intrinsics and mapper settings.