A Study of Finetuning Video Transformers for Multi-view Geometry Tasks

TL;DR

GeoViT demonstrates that general-purpose video-pretrained Vision Transformers can be effectively finetuned for multi-view geometry tasks with minimal adaptation, challenging the need for task-specific pretraining and complex architectures. By adapting two-frame inputs through positional embeddings and employing a simple linear decoder, optionally augmented by iterative refinement, the method achieves state-of-the-art cross-dataset performance in optical flow and strong results in stereo and depth estimation. Across optical flow, stereo matching, and depth estimation, GeoViT shows robust generalization and competitive benchmarks on Sintel, KITTI, ETH3D, SUN3D, RGBD-SLAM, and Scene Flow datasets. These findings highlight the potential of video pretraining to capture temporal and spatial cues useful for precise geometric reasoning, suggesting a scalable pathway for geometric vision using video foundation models.

Abstract

This paper presents an investigation of vision transformer learning for multi-view geometry tasks, such as optical flow estimation, by fine-tuning video foundation models. Unlike previous methods that involve custom architectural designs and task-specific pretraining, our research finds that general-purpose models pretrained on videos can be readily transferred to multi-view problems with minimal adaptation. The core insight is that general-purpose attention between patches learns temporal and spatial information for geometric reasoning. We demonstrate that appending a linear decoder to the Transformer backbone produces satisfactory results, and iterative refinement can further elevate performance to stateof-the-art levels. This conceptually simple approach achieves top cross-dataset generalization results for optical flow estimation with end-point error (EPE) of 0.69, 1.78, and 3.15 on the Sintel clean, Sintel final, and KITTI datasets, respectively. Our method additionally establishes a new record on the online test benchmark with EPE values of 0.79, 1.88, and F1 value of 3.79. Applications to 3D depth estimation and stereo matching also show strong performance, illustrating the versatility of video-pretrained models in addressing geometric vision tasks.

Figures (2)

• Figure 1: Overview of GeoViT. Part (a) presents adaptation of positional embeddings in pretrained 3D ViTs. The pretrained spatial Pos. Embd. (in green) are interpolated to match the desired input size. The pretrained temporal Pos. Embd. (in purple), which accounts for 8 frames, is split into two halves. The average of the first half and second half corresponds to the temporal embedding of the source image (in orange) and target image (in blue), respectively. Part (b) exhibits our iterative refinement decoding pipeline, using optical flow for illustration. The input target image is dynamically warped based on the last-step prediction $g_{t-1}$ so that the input pair corresponds to the ground truth residual for this step. Then the source and (warped) target images are patchified, added with adapted positional embeddings, and fed to the pretrained 3D ViT for feature extraction. The decoder accepts source image features (in light orange) and the last-step prediction ($g_{t-1}$) and produces its correction $\Delta g_t$. Adding $g_{t-1}$ and its correction gives the current-step prediction $g_{t}$.
• Figure 2: Visualized prediction comparison on Sintel (clean) dataset. Our approach is more accurate with more fine-grained estimates (human shoulder region in case #1), a higher recall of small objects (bird region in case #2), and a crisper motion boundary (case #2). The highlighted region is zoomed in for better visual comparison.