FreeOrbit4D: Training-Free Arbitrary Camera Redirection for Monocular Videos via Geometry-Complete 4D Reconstruction

TL;DR

This work tackles camera redirection from a single monocular video, a severely ill-posed problem under large-angle viewpoints. It introduces FreeOrbit4D, a training-free framework that constructs a geometry-complete 4D proxy by decoupling global scene lifting (background + incomplete foreground) from canonical object completion (foreground via multi-view synthesis), aligning these into a unified proxy with dense 3D correspondences, and conditioning a depth-guided diffusion model for novel-view synthesis. The approach yields superior fidelity and temporal coherence for large-angle redirects, outperforming state-of-the-art methods and enabling applications like appearance propagation and 4D geometry editing. By providing explicit 4D geometry, FreeOrbit4D offers robust re-visualization, editability, and data-generation potential for 4D scenes, all without training on paired data.

Abstract

Camera redirection aims to replay a dynamic scene from a single monocular video under a user-specified camera trajectory. However, large-angle redirection is inherently ill-posed: a monocular video captures only a narrow spatio-temporal view of a dynamic 3D scene, providing highly partial observations of the underlying 4D world. The key challenge is therefore to recover a complete and coherent representation from this limited input, with consistent geometry and motion. While recent diffusion-based methods achieve impressive results, they often break down under large-angle viewpoint changes far from the original trajectory, where missing visual grounding leads to severe geometric ambiguity and temporal inconsistency. To address this, we present FreeOrbit4D, an effective training-free framework that tackles this geometric ambiguity by recovering a geometry-complete 4D proxy as structural grounding for video generation. We obtain this proxy by decoupling foreground and background reconstructions: we unproject the monocular video into a static background and geometry-incomplete foreground point clouds in a unified global space, then leverage an object-centric multi-view diffusion model to synthesize multi-view images and reconstruct geometry-complete foreground point clouds in canonical object space. By aligning the canonical foreground point cloud to the global scene space via dense pixel-synchronized 3D--3D correspondences and projecting the geometry-complete 4D proxy onto target camera viewpoints, we provide geometric scaffolds that guide a conditional video diffusion model. Extensive experiments show that FreeOrbit4D produces more faithful redirected videos under challenging large-angle trajectories, and our geometry-complete 4D proxy further opens a potential avenue for practical applications such as edit propagation and 4D data generation. Project page and code will be released soon.

Figures (9)

• Figure 1: Comparison of video camera redirection paradigms. We compare $3$ representative approaches for camera redirection from monocular video. (A) Implicit Control: Camera motion is specified via learned embeddings. Such implicit representations provide only soft controllability: text cannot precisely describe complex trajectories, and learned conditions often fail to follow the intended path (e.g., the "turn to back" instruction). Moreover, training requires paired data, which is expensive and scarce. (B) Explicit Warping: These methods warp observed pixels to target viewpoints using estimated depth. However, occluded regions remain unfilled, producing visible holes (red circles show "Half Camel"). (C) Ours: We reconstruct a geometry-complete 4D proxy that recovers both visible and occluded surfaces, then render it from target viewpoints as geometric guidance for video generation. This enables precise camera control with complete visibility from arbitrary viewpoints (green circle shows "Complete Camel"). The table below summarizes key trade-offs across camera controllability and visibility completeness. $\checkmark$, and $\times$ denote full and lack of support.
• Figure 2: Overview of FreeOrbit4D. Our framework redirects a monocular video to a target camera trajectory via a geometry-complete 4D proxy. This proxy is constructed through two branches: Global Scene Reconstruction recovers background and partial foreground in global space, while Canonical Object Completion reconstructs complete foreground geometry via multi-view synthesis. After alignment, we render view-dependent depth maps from the unified 4D proxy and condition a video diffusion model, along with a reference image and text prompt, enabling faithful novel-view synthesis under large viewpoint changes.
• Figure 3: Decoupled 4D reconstruction and alignment pipeline.Left (Sec. \ref{['subsec:reconstruction']}): A dynamic-aware feed-forward network lifts the source video $\mathcal{V}^{src}$ into global scene space, producing the static background $\mathcal{P}^{bg}$ and geometry-incomplete foreground $\widetilde{\mathcal{P}}_t^{fg}$ (orange). Middle (Sec. \ref{['subsec:reconstruction']}): The foreground sequence $\{\mathbf{I}_t^{fg}\}_{t=1}^T$ is fed into an object-centric video diffusion model to synthesize multi-view images, from which VGGT reconstructs geometry-complete foreground $\widehat{\mathcal{P}}_t^{fg}$ in canonical object space. Right (Sec. \ref{['subsec:alignment']}): Dense 3D-3D correspondences derived from pixel-synchronized point maps enable correspondence-aware alignment: per-frame spatial alignment estimates scale $s_t$ and translation $\mathbf{t}_t$, followed by temporal smoothing via Kalman filtering. The result is the unified geometry-complete 4D proxy $\mathcal{P} = \mathcal{P}^{bg} \cup \{\mathcal{P}_t^{fg}\}_{t=1}^T$.
• Figure 4: Qualitative comparison on challenging dynamic sequences: "Swing" (top) and "Bmx" (bottom). For each sequence, the top-left inset visualizes our reconstructed 4D geometry-complete proxy and the target camera trajectory. These scenarios feature rapid foreground motion, thin structures (e.g., swing ropes), and significant perspective changes. Existing methods exhibit distinct failure modes: ReCamMaster 25iccv/recammaster and EX4D 25arxiv/ex4d suffer from structural disintegration, failing to preserve the integrity of fast-moving subjects (e.g., blurred limbs and ghosting). In contrast, TrajectoryCrafter 25iccv/trajectorycrafter and GEN3C 25cvpr/gen3c tend to produce geometric warping and semantic drift, where the background and subjects undergo unnatural distortions as the viewpoint evolves. Our method consistently generates sharp details and stable 4D geometry, effectively handling complex interactions and disocclusions by anchoring pixels to the explicit geometry-complete proxy.
• Figure 5: Applications enabled by FreeOrbit4D. Our explicit 4D representation enables various downstream applications. Top: Appearance editing---given a single edited reference frame (e.g., zebra pattern or anime style), our geometry-complete proxy propagates the edit consistently across all novel viewpoints. Bottom: Geometry editing---by directly manipulating the point cloud (scaling or compositing objects from different sources), we synthesize plausible redirected videos from the modified 4D geometry.