PoseCrafter: Extreme Pose Estimation with Hybrid Video Synthesis

TL;DR

PoseCrafter targets extreme pose estimation when image pairs have limited overlap by synthesizing informative intermediate views using Hybrid Video Generation (HVG) and selecting frames via a deterministic Feature Matching Selector (FMS). HVG blends motion-driven interpolation with pose-conditioned novel-view synthesis to produce high-fidelity frames that preserve geometry, while FMS ensures only frames with strong geometric consistency are used for final pose estimation. The approach is training-free and achieves state-of-the-art performance across four benchmarks (Cambridge Landmarks, ScanNet, DL3DV-10K, NAVI), particularly in small-overlap scenarios, with favorable runtime compared to self-consistency-based methods. Ablation studies corroborate the benefits of HVG, the choice of frame-count, and the reliability of ORB-based matching within FMS. The work demonstrates practical impact for robust pose estimation in challenging real-world scenes and broadens the applicability of diffusion-based video synthesis to downstream geometric tasks.

Abstract

Pairwise camera pose estimation from sparsely overlapping image pairs remains a critical and unsolved challenge in 3D vision. Most existing methods struggle with image pairs that have small or no overlap. Recent approaches attempt to address this by synthesizing intermediate frames using video interpolation and selecting key frames via a self-consistency score. However, the generated frames are often blurry due to small overlap inputs, and the selection strategies are slow and not explicitly aligned with pose estimation. To solve these cases, we propose Hybrid Video Generation (HVG) to synthesize clearer intermediate frames by coupling a video interpolation model with a pose-conditioned novel view synthesis model, where we also propose a Feature Matching Selector (FMS) based on feature correspondence to select intermediate frames appropriate for pose estimation from the synthesized results. Extensive experiments on Cambridge Landmarks, ScanNet, DL3DV-10K, and NAVI demonstrate that, compared to existing SOTA methods, PoseCrafter can obviously enhance the pose estimation performances, especially on examples with small or no overlap.

Figures (6)

• Figure 1: Overview of the PoseCrafter pipeline. Starting from an input image pair (left), the Hybrid Video Generation module uses DynamiCrafter to interpolate a short video clip, extracts the most reliable “relay” frames, and refines the sequence using the pose-conditioned ViewCrafter to reduce mid-sequence blur and drift. The Feature Matching Selector then evaluates each synthesized frame based on its feature correspondence with the input pair, selecting the most informative views for pose estimation. These selected frames are then fed into a pose estimation model (e.g. DUSt3R) to estimate the final pose.
• Figure 2: Examples of mid-frame blurriness and confidence degradation. (a) presents frames generated by DynamiCrafter (DC) and our proposed method. (b) highlights high-confidence regions identified by DUST3R during pose estimation. Green boxes mark areas affected by blurriness in the DC-generated frames, while red boxes indicate how these regions performed in DUST3R’s confidence maps. We observe that the corresponding regions in the DynamiCrafter results exhibit notably low confidence, whereas our method generates frames with higher confidence for DUST3R.
• Figure 3: Qualitative comparisons on common benchmarks. Blue and yellow outlines indicate the start and end frames, respectively. In the DUSt3R reconstruction results, the visualized poses in blue, yellow, and red represent the predicted pose of the start frame, predicted pose of the end frame, and the ground-truth pose of the end frame, respectively.
• Figure 4: Hybrid Video Generation artifacts under severe illumination differences. Blue and yellow outlines indicate the start and end frames, respectively. Confidence images denote images filtered with the predicted confidence map in the subsequent DUSt3R model. The red boxes highlight regions affected by artifacts in these cases. We can observe that these regions have quite low confidence.
• Figure 5: Feature Matching Selection failures in low-texture regions. Blue and yellow outlines indicate the start and end frames, respectively. The red and green lines indicate incorrect and correct correspondences, respectively. Incorrect correspondences lead to errors in inlier counting, affecting the accuracy of subsequent frame selection.