TALO: Pushing 3D Vision Foundation Models Towards Globally Consistent Online Reconstruction

TL;DR

3D Vision Foundation Models excel offline but struggle to maintain online temporal consistency in driving-like sequences due to spatially varying distortions. The paper analyzes the limitations of global-warp alignments (Sim(3) and SL(4)) and introduces TALO, a higher-DOF TPS-based online alignment with globally propagated control points and a point-agnostic submap registration. By aggregating multi-view control-point observations and applying a TPS warp to a global canonical space, TALO achieves more coherent geometry and lower trajectory errors across diverse datasets and camera configurations, without relying on a single global transform. The results demonstrate robustness and generality across backbone models and setups, offering practical benefits for online 3D reconstruction in dynamic scenes such as driving.

Abstract

3D vision foundation models have shown strong generalization in reconstructing key 3D attributes from uncalibrated images through a single feed-forward pass. However, when deployed in online settings such as driving scenarios, predictions are made over temporal windows, making it non-trivial to maintain consistency across time. Recent strategies align consecutive predictions by solving global transformation, yet our analysis reveals their fundamental limitations in assumption validity, local alignment scope, and robustness under noisy geometry. In this work, we propose a higher-DOF and long-term alignment framework based on Thin Plate Spline, leveraging globally propagated control points to correct spatially varying inconsistencies. In addition, we adopt a point-agnostic submap registration design that is inherently robust to noisy geometry predictions. The proposed framework is fully plug-and-play, compatible with diverse 3D foundation models and camera configurations (e.g., monocular or surround-view). Extensive experiments demonstrate that our method consistently yields more coherent geometry and lower trajectory errors across multiple datasets, backbone models, and camera setups, highlighting its robustness and generality. Codes are publicly available at \href{https://github.com/Xian-Bei/TALO}{https://github.com/Xian-Bei/TALO}.

Figures (8)

• Figure 1: Degeneration of $\mathrm{Sim}(3)$ alignment used in VGGT-Long vggtlong and $\mathrm{SL}(4)$ in VGGT-SLAM vggtslam. Two consecutive submaps (left) independently predicted by a foundation model exhibit spatially varying, nonlinear geometric inconsistencies. Since neither $\mathrm{Sim}(3)$ nor $\mathrm{SL}(4)$ can theoretically reconcile such non-global distortions with a single global transformation, enforcing a global warp inevitably overfits one region at the expense of another, leaving noticeable alignment residuals (visualized by colored connecting lines and zoomed-in on the right, where severe wall ghosting occurs). Moreover, the under-constrained $\mathrm{SL}(4)$ is highly sensitive to noise and often yields divergent geometry (e.g., severely tilted buildings, blue circle) and physically implausible camera poses. As shown in the orange box, the three cameras predicted by the foundation model on the right maintain consistent forward-facing orientations, whereas after $\mathrm{SL}(4)$ alignment, their pitch angles diverge drastically, resulting in an impossible trajectory in driving scenarios.
• Figure 2: Workflow of TALO. TALO processes multi-camera continuous sequences by first dividing them into submaps with overlapping frames. It then performs point-agnostic submap registration by averaging the relative transformations of the overlapping frames, followed by generating control points within the overlapping regions. These control points are globally propagated across submaps, and all their observations are finally aggregated to construct a TPS deformation field that warps every submap into a globally consistent canonical space.
• Figure 3: Qualitative comparison of the reconstructed trajectories and geometries with $\pi^3$pi3 as backbone on nuScenesnuScenes scene-0039 (left) and Waymowaymo scene 16345319168590318167 (right), respectively. More results are shown in the supplementary materials.
• Figure 4: Effect of camera count and submap size on trajectory and geometry accuracy. Across all settings, TALO consistently achieves the lowest ATE and Chamfer Distance. Larger submaps reduce trajectory error, while additional cameras improve geometric stability, confirming that richer viewpoints strengthen VFM–based estimation.
• Figure 5: Qualitative comparison with VGGT vggt on Waymowaymo scene 6104545334635651714.