QuaDreamer: Controllable Panoramic Video Generation for Quadruped Robots

TL;DR

QuaDreamer introduces the first panoramic data generation engine for quadruped robots, combining Vertical Jitter Encoding, Scene-Object Controller, and Panoramic Enhancer to produce controllable 360-degree videos that capture gait-induced vertical jitter and panoramic distortions. The approach uses a high-pass filter to extract jitter, a temporal-attention-based SOC with Fourier embeddings to fuse motion signals, and a dual-stream PE with SSM and Fourier convolutions to improve global consistency. Empirical results on QuadTrack show improvements in perceptual quality (LPIPS, SSIM, PSNR) and controllability (PTrack), and downstream data augmentation enhances multi-object tracking metrics like HOTA and MOTA. The work demonstrates practical utility for perception in complex, open environments and provides open-source code to advance quadruped vision research.

Abstract

Panoramic cameras, capturing comprehensive 360-degree environmental data, are suitable for quadruped robots in surrounding perception and interaction with complex environments. However, the scarcity of high-quality panoramic training data-caused by inherent kinematic constraints and complex sensor calibration challenges-fundamentally limits the development of robust perception systems tailored to these embodied platforms. To address this issue, we propose QuaDreamer-the first panoramic data generation engine specifically designed for quadruped robots. QuaDreamer focuses on mimicking the motion paradigm of quadruped robots to generate highly controllable, realistic panoramic videos, providing a data source for downstream tasks. Specifically, to effectively capture the unique vertical vibration characteristics exhibited during quadruped locomotion, we introduce Vertical Jitter Encoding (VJE). VJE extracts controllable vertical signals through frequency-domain feature filtering and provides high-quality prompts. To facilitate high-quality panoramic video generation under jitter signal control, we propose a Scene-Object Controller (SOC) that effectively manages object motion and boosts background jitter control through the attention mechanism. To address panoramic distortions in wide-FoV video generation, we propose the Panoramic Enhancer (PE)-a dual-stream architecture that synergizes frequency-texture refinement for local detail enhancement with spatial-structure correction for global geometric consistency. We further demonstrate that the generated video sequences can serve as training data for the quadruped robot's panoramic visual perception model, enhancing the performance of multi-object tracking in 360-degree scenes. The source code and model weights will be publicly available at https://github.com/losehu/QuaDreamer.

Figures (9)

• Figure 1: Illustration of the proposed QuaDreamer, the first panoramic video generation framework tailored for quadruped robots. QuaDreamer enables generation with control via box and jitter prompts, serving as a data source to enhance the performance of downstream tasks.
• Figure 2: The overall framework of our QuaDreamer. Vertical Jitter Encoding extracts jitter signals from bounding boxes and combines them with box information to accurately model motion patterns. To further enhance realism, we incorporate a Scene-Object Controller and a Panoramic Enhancer, which jointly manage object dynamics and refine the representation of panoramic motion.
• Figure 3: Illustration of the control components, including VJE and SOC. (a) shows the original y-axis pixel coordinate data and its low-frequency component; (b) displays the filtered high-frequency jitter data; (c) illustrates the frequency spectrum of the original data.
• Figure 4: Visualization results generated on the QuadTrack dataset OminiTrack. The rainbow-colored trajectory is the CoTracker karaev2024cotracker3 jittering trajectory. The trajectory in the red boxes clearly demonstrates that our jitter control is more similar to the ground truth.
• Figure 5: Reasoning for simulated yaw angle changes