L4P: Towards Unified Low-Level 4D Vision Perception

TL;DR

L4P introduces a unified, feedforward architecture that leverages a pre-trained VideoMAE encoder to tackle both dense and sparse low-level 4D vision perception tasks within a single framework. By pairing VideoMAE with lightweight, task-specific heads and a novel memory mechanism for long-range tracking, the model achieves competitive results across depth, optical flow, tracking, motion segmentation, and camera pose estimation while maintaining efficiency (≈300ms for a 16-frame clip). The approach demonstrates strong generalization to real-world data through a careful training curriculum on synthetic datasets and shows extensibility to additional tasks without sacrificing existing performance. This work highlights the potential of video-prior-based foundation models to unify diverse 4D perception tasks and enable real-time-like, multi-task reasoning in dynamic scenes.

Abstract

The spatio-temporal relationship between the pixels of a video carries critical information for low-level 4D perception tasks. A single model that reasons about it should be able to solve several such tasks well. Yet, most state-of-the-art methods rely on architectures specialized for the task at hand. We present L4P, a feedforward, general-purpose architecture that solves low-level 4D perception tasks in a unified framework. L4P leverages a pre-trained ViT-based video encoder and combines it with per-task heads that are lightweight and therefore do not require extensive training. Despite its general and feedforward formulation, our method is competitive with existing specialized methods on both dense tasks, such as depth or optical flow estimation, and sparse tasks, such as 2D/3D tracking. Moreover, it solves all tasks at once in a time comparable to that of single-task methods.

Figures (7)

• Figure 1: We propose L4P, a general-purpose architecture that solves several low-level 4D perception tasks. Building on a pre-trained video encoder and lightweight per-task heads, our unified model is competitive with existing methods specialized to solve individual tasks. L4P can easily be extended to additional tasks, (e) and (f), without compromising performance.
• Figure 2: We use a pre-trained video encoder as tokenizer, and combine it with per-task lightweight heads. For sparse tasks we define additional query tokens: the point to track, $\mathcal{P}$, the corresponding feature token, $\mathcal{F}$, and output tokens (heatmap $\mathcal{H}$, depth $\mathcal{D}$, and visibility $\mathcal{V}$).
• Figure 3: Sparse head. The video and I/O tokens (query $\mathcal{P}$oint, query point $\mathcal{F}$eature, and query point $\mathcal{H}$eat map, $\mathcal{D}$epth, and $\mathcal{V}$isibility) are processed by a two-way attention layer. The reshaped and resized per-frame feature maps and processed output tokens are combined via a dot product. We also introduce a memory mechanism to combine video and the $\mathcal{F}$eature tokens from time $t$ and $t+1$.
• Figure 4: Comparisons with SOTA methods. We run inference on videos but show one frame here. The purple masks show GT motion segmentation.
• Figure 5: Comparisons on 2D/3D tracking (a). L4P can estimate camera poses, depth, and tracking in the same reference system (b).