Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DATAP-SfM: Dynamic-Aware Tracking Any Point for Robust Structure from Motion in the Wild

Weicai Ye, Xinyu Chen, Ruohao Zhan, Di Huang, Xiaoshui Huang, Haoyi Zhu, Hujun Bao, Wanli Ouyang, Tong He, Guofeng Zhang

TL;DR

This paper proposes a dynamic-aware tracking any point (DATAP) method that leverages consistent video depth and point tracking and achieves state-of-the-art performance in terms of camera pose estimation even in complex dynamic challenge scenes.

Abstract

This paper proposes a concise, elegant, and robust pipeline to estimate smooth camera trajectories and obtain dense point clouds for casual videos in the wild. Traditional frameworks, such as ParticleSfM~\cite{zhao2022particlesfm}, address this problem by sequentially computing the optical flow between adjacent frames to obtain point trajectories. They then remove dynamic trajectories through motion segmentation and perform global bundle adjustment. However, the process of estimating optical flow between two adjacent frames and chaining the matches can introduce cumulative errors. Additionally, motion segmentation combined with single-view depth estimation often faces challenges related to scale ambiguity. To tackle these challenges, we propose a dynamic-aware tracking any point (DATAP) method that leverages consistent video depth and point tracking. Specifically, our DATAP addresses these issues by estimating dense point tracking across the video sequence and predicting the visibility and dynamics of each point. By incorporating the consistent video depth prior, the performance of motion segmentation is enhanced. With the integration of DATAP, it becomes possible to estimate and optimize all camera poses simultaneously by performing global bundle adjustments for point tracking classified as static and visible, rather than relying on incremental camera registration. Extensive experiments on dynamic sequences, e.g., Sintel and TUM RGBD dynamic sequences, and on the wild video, e.g., DAVIS, demonstrate that the proposed method achieves state-of-the-art performance in terms of camera pose estimation even in complex dynamic challenge scenes.

DATAP-SfM: Dynamic-Aware Tracking Any Point for Robust Structure from Motion in the Wild

TL;DR

This paper proposes a dynamic-aware tracking any point (DATAP) method that leverages consistent video depth and point tracking and achieves state-of-the-art performance in terms of camera pose estimation even in complex dynamic challenge scenes.

Abstract

This paper proposes a concise, elegant, and robust pipeline to estimate smooth camera trajectories and obtain dense point clouds for casual videos in the wild. Traditional frameworks, such as ParticleSfM~\cite{zhao2022particlesfm}, address this problem by sequentially computing the optical flow between adjacent frames to obtain point trajectories. They then remove dynamic trajectories through motion segmentation and perform global bundle adjustment. However, the process of estimating optical flow between two adjacent frames and chaining the matches can introduce cumulative errors. Additionally, motion segmentation combined with single-view depth estimation often faces challenges related to scale ambiguity. To tackle these challenges, we propose a dynamic-aware tracking any point (DATAP) method that leverages consistent video depth and point tracking. Specifically, our DATAP addresses these issues by estimating dense point tracking across the video sequence and predicting the visibility and dynamics of each point. By incorporating the consistent video depth prior, the performance of motion segmentation is enhanced. With the integration of DATAP, it becomes possible to estimate and optimize all camera poses simultaneously by performing global bundle adjustments for point tracking classified as static and visible, rather than relying on incremental camera registration. Extensive experiments on dynamic sequences, e.g., Sintel and TUM RGBD dynamic sequences, and on the wild video, e.g., DAVIS, demonstrate that the proposed method achieves state-of-the-art performance in terms of camera pose estimation even in complex dynamic challenge scenes.

Paper Structure

This paper contains 16 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Dynamic-Aware Tracking Any Point (DATAP)
  5. Structure from Motion with DATAP
  6. Experiment
  7. Implementation Details
  8. Dataset
  9. ScanNet.
  10. Evaluation metrics
  11. Comparison on MPI Sintel dataset
  12. Comparison on TUM RGBD Dynamic dataset
  13. Qualitative evaluation on in-the-wild videos
  14. Ablation Study
  15. Conclusion
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Given casual videos, our method can obtain smooth camera trajectories and entire point clouds of dynamic scenes. From top to bottom: video samples, results from COLMAP, ParticleSfM, and ours.
  • Figure 2: DATAP-SfM pipeline. Given monocular videos as input with consistent video depth (optional), DATAP can simultaneously estimate long-term point tracking with visible and dynamic characteristics. Incorporating with DATAP, we formalize a concise, elegant, and robust pipeline of structure from motion by performing global bundle adjustment for point tracking classified as static and visible.
  • Figure 3: Qualitative results of motion segmentation on MPI Sintel dataset. Our method outperforms existing SOTA methods. From top to bottom: image samples, motion segmentation results from Oneformer jain2023oneformer, ParticleSfM, and ours. Red: static, green: dynamic. The third column is the sleeping case, which should be static.
  • Figure 4: Qualitative results of camera pose estimation on MPI Sintel dataset. Our method outperforms existing SOTA methods.
  • Figure 5: Qualitative results of structure from motion on DAVIS dataset. Our method can obtain smooth camera trajectories and entire point clouds of dynamic scenes.