Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Encoder-Based 3D from Casual Videos via Point Track Processing

Yoni Kasten, Wuyue Lu, Haggai Maron

TL;DR

TracksTo4D presents a fast, learning-based solution to recover dynamic 3D structure and camera motion from casual videos by operating on 2D point tracks. The method employs symmetry-aware equivariant transformers and a low-rank, basis-based representation to constrain the problem, enabling unsupervised training from 2D reprojection losses. It achieves comparable accuracy to state-of-the-art methods while providing substantial speedups (up to 95% faster) and strong generalization to unseen video content. This approach enables practical 3D reconstruction from everyday videos without per-video optimization or 3D supervision. The combination of point-track processing, equivariant design, and static/dynamic decomposition broadens the applicability of dynamic scene reconstruction in real-world settings.

Abstract

This paper addresses the long-standing challenge of reconstructing 3D structures from videos with dynamic content. Current approaches to this problem were not designed to operate on casual videos recorded by standard cameras or require a long optimization time. Aiming to significantly improve the efficiency of previous approaches, we present TracksTo4D, a learning-based approach that enables inferring 3D structure and camera positions from dynamic content originating from casual videos using a single efficient feed-forward pass. To achieve this, we propose operating directly over 2D point tracks as input and designing an architecture tailored for processing 2D point tracks. Our proposed architecture is designed with two key principles in mind: (1) it takes into account the inherent symmetries present in the input point tracks data, and (2) it assumes that the movement patterns can be effectively represented using a low-rank approximation. TracksTo4D is trained in an unsupervised way on a dataset of casual videos utilizing only the 2D point tracks extracted from the videos, without any 3D supervision. Our experiments show that TracksTo4D can reconstruct a temporal point cloud and camera positions of the underlying video with accuracy comparable to state-of-the-art methods, while drastically reducing runtime by up to 95\%. We further show that TracksTo4D generalizes well to unseen videos of unseen semantic categories at inference time.

Fast Encoder-Based 3D from Casual Videos via Point Track Processing

TL;DR

TracksTo4D presents a fast, learning-based solution to recover dynamic 3D structure and camera motion from casual videos by operating on 2D point tracks. The method employs symmetry-aware equivariant transformers and a low-rank, basis-based representation to constrain the problem, enabling unsupervised training from 2D reprojection losses. It achieves comparable accuracy to state-of-the-art methods while providing substantial speedups (up to 95% faster) and strong generalization to unseen video content. This approach enables practical 3D reconstruction from everyday videos without per-video optimization or 3D supervision. The combination of point-track processing, equivariant design, and static/dynamic decomposition broadens the applicability of dynamic scene reconstruction in real-world settings.

Abstract

This paper addresses the long-standing challenge of reconstructing 3D structures from videos with dynamic content. Current approaches to this problem were not designed to operate on casual videos recorded by standard cameras or require a long optimization time. Aiming to significantly improve the efficiency of previous approaches, we present TracksTo4D, a learning-based approach that enables inferring 3D structure and camera positions from dynamic content originating from casual videos using a single efficient feed-forward pass. To achieve this, we propose operating directly over 2D point tracks as input and designing an architecture tailored for processing 2D point tracks. Our proposed architecture is designed with two key principles in mind: (1) it takes into account the inherent symmetries present in the input point tracks data, and (2) it assumes that the movement patterns can be effectively represented using a low-rank approximation. TracksTo4D is trained in an unsupervised way on a dataset of casual videos utilizing only the 2D point tracks extracted from the videos, without any 3D supervision. Our experiments show that TracksTo4D can reconstruct a temporal point cloud and camera positions of the underlying video with accuracy comparable to state-of-the-art methods, while drastically reducing runtime by up to 95\%. We further show that TracksTo4D generalizes well to unseen videos of unseen semantic categories at inference time.
Paper Structure (42 sections, 10 equations, 4 figures, 4 tables)

This paper contains 42 sections, 10 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Previous work and limitations.
  3. Our approach.
  4. Contributions.
  5. Method
  6. Problem formulation.
  7. Overview of our approach.
  8. Equivariant layers for point track tensors
  9. Symmetry analysis.
  10. Linear equivariant layers.
  11. Implementation via transformers and positional encoding.
  12. Constraining 3D motion and camera poses via low-rank assumption
  13. Low-rank movement assumption.
  14. Low-rank movement assumption.
  15. Specific modeling of the static scene for camera estimation.
  16. ...and 27 more sections

Figures (4)

  • Figure 1: We present TracksTo4D, a method for mapping a set of 2D point tracks extracted from casual dynamic videos into their corresponding 3D locations and camera motion. At inference time, our network predicts the dynamic structure and camera motion in a single feed-forward pass. Our network takes as input a set of 2D point tracks (left) and uses several multi-head attention layers while alternating between the time dimension and the track dimension (middle). The network predicts cameras, per-frame 3D points, and per-world point movement value (right). The 3D point internal colors illustrate the predicted 3D movement level values, such that points with high/low 3D motion are presented in red/purple colors respectively. These outputs are used to reproject the predicted points into the frames for calculating the reprojection error losses. See details in the text. The reader is encouraged to watch the supplementary video visualizations.
  • Figure 2: The symmetry structure of our problem. Frames (vertical) have time translation symmetry while points (horizontal) have set permutation symmetry.
  • Figure 3: Qualitative Results. Top. Frames from 2 different test video sequences with point tracks marked with corresponding colors. Bottom. A 3D visualization of our method's outputs, from two time stamps. The camera trajectory is present as gray frustums, whereas the current camera is marked in red. The reconstructed 3D scene points are presented in corresponding colors to the input tracks on the top. The scene is observed from the same viewpoint, enabling the visualization of the dynamic reconstructed structure.
  • Figure 4: $\gamma$ Visualization. We show a visualization of the $\gamma$ outputs of our network that are described in Sec. \ref{['section::output_parameterization']}. In each video sequence, we show the input tracks, where each color visualizes its movement level value, $\gamma$. Purple marks static points with low $\gamma$ whereas red marks dynamic points with high $\gamma$. Note, that our network did not get any direct supervision for these values, but only the raw point tracks predictions from karaev2023cotracker. The $\gamma$ visualizations for cats were produced by the model that was only trained on dogs and vice versa. We note that our model generalizes well to out-of-domain (non-pet) cases as well.