GimbalDiffusion: Gravity-Aware Camera Control for Video Generation

TL;DR

GimbalDiffusion introduces gravity-aligned absolute camera control for text-to-video generation by grounding camera orientation to a global gravity reference. It builds a data pipeline from geometrically calibrated 360° panoramas to sample diverse spherical trajectories, and employs null-pitch conditioning to disentangle text from camera angle. A new SpatialVID-extreme benchmark evaluates performance under extreme pitches and rolls, and experiments show substantial improvements in absolute pitch and gravity accuracy with robust, interpretable camera motion. The work advances physically grounded controllability in diffusion-based video synthesis and lays groundwork for future translation-aware extensions and artifact mitigation.

Abstract

Recent progress in text-to-video generation has achieved remarkable realism, yet fine-grained control over camera motion and orientation remains elusive. Existing approaches typically encode camera trajectories through relative or ambiguous representations, limiting explicit geometric control. We introduce GimbalDiffusion, a framework that enables camera control grounded in physical-world coordinates, using gravity as a global reference. Instead of describing motion relative to previous frames, our method defines camera trajectories in an absolute coordinate system, allowing precise and interpretable control over camera parameters without requiring an initial reference frame. We leverage panoramic 360-degree videos to construct a wide variety of camera trajectories, well beyond the predominantly straight, forward-facing trajectories seen in conventional video data. To further enhance camera guidance, we introduce null-pitch conditioning, an annotation strategy that reduces the model's reliance on text content when conflicting with camera specifications (e.g., generating grass while the camera points towards the sky). Finally, we establish a benchmark for camera-aware video generation by rebalancing SpatialVID-HQ for comprehensive evaluation under wide camera pitch variation. Together, these contributions advance the controllability and robustness of text-to-video models, enabling precise, gravity-aligned camera manipulation within generative frameworks.

Figures (6)

• Figure 1: Reproducing real scenes using our representation. From reference frames from a driving video (left), GimbalDiffusion can be controlled to replicate its camera viewpoint with respect to gravity (center). In contrast, existing methods (right) do not accurately match the absolute reference pose.
• Figure 2: Training data pipeline. (a) We extract the camera poses from a dataset of 360° panoramic videos in equirectangular format (see \ref{['sec:dataset']}). (b) We randomly sample a camera rotation trajectory on the sphere, from which we generate field of view masks in equirectangular format for both the projected image and the null-pitch conditioning (see \ref{['sec:nullpitch']}). (c) We integrate the two previous steps by combining the sampled rotation with the estimated camera poses. This process creates a dataset of perspective images with their camera poses, along with null-pitch reference images. The main advantage of this approach is the ability to sample camera viewing directions across the entire sphere, in contrast to using natural human-captured videos, which are heavily biased towards null roll and pitch. Additionally, we generate a text description for both the perspective video and the null-pitch video using a VLM.
• Figure 3: Training data samples from our data augmentation pipeline, capturing a highly diverse set of rotation trajectories from 360° videos. Both the trajectory and the prompt are generated on-the-fly in our dataloader. Each color in the graph corresponds to a video above.
• Figure 4: Comparison of parameter distribution between our sampling and a typical video dataset for camera control (RealEstate10K RealEstate10K). Our sampling produces videos with more rotational motion, as measured by total angular distance (the sum of absolute angular differences between consecutive camera poses), and greater diversity in Euler angles (pitch, roll, yaw). All values on the $x$-axis are expressed in degrees.
• Figure 5: Entanglement between prompt and camera pitch. Without a careful captioning strategy, diffusion models may ignore the camera conditioning when the prompt semantics conflict with it. Our null-pitch conditioning mitigates this issue and preserves accurate camera control.