360Anything: Geometry-Free Lifting of Images and Videos to 360°

TL;DR

360Anything addresses the problem of converting perspective images and videos into 360° panoramas without camera calibration. It adopts a geometry-free, sequence-based diffusion Transformer that learns the mapping from perspective inputs to equirectangular panoramas by treating both as latent token sequences, thus avoiding explicit projection and intrinsics. The authors identify VAE latent padding as the seam source and introduce Circular Latent Encoding to achieve seam-free generation, while enforcing a Canonical Coordinate training objective to produce upright panoramas. The approach delivers state-of-the-art results for both image and video perspective-to-360° generation, demonstrates strong zero-shot FoV and pose estimation capabilities, and supports downstream 3D scene reconstruction, highlighting its practical impact for in-the-wild panorama synthesis.

Abstract

Lifting perspective images and videos to 360° panoramas enables immersive 3D world generation. Existing approaches often rely on explicit geometric alignment between the perspective and the equirectangular projection (ERP) space. Yet, this requires known camera metadata, obscuring the application to in-the-wild data where such calibration is typically absent or noisy. We propose 360Anything, a geometry-free framework built upon pre-trained diffusion transformers. By treating the perspective input and the panorama target simply as token sequences, 360Anything learns the perspective-to-equirectangular mapping in a purely data-driven way, eliminating the need for camera information. Our approach achieves state-of-the-art performance on both image and video perspective-to-360° generation, outperforming prior works that use ground-truth camera information. We also trace the root cause of the seam artifacts at ERP boundaries to zero-padding in the VAE encoder, and introduce Circular Latent Encoding to facilitate seamless generation. Finally, we show competitive results in zero-shot camera FoV and orientation estimation benchmarks, demonstrating 360Anything's deep geometric understanding and broader utility in computer vision tasks. Additional results are available at https://360anything.github.io/.

Figures (15)

• Figure 1: 360Anything lifts arbitrary perspective images (row 1) and videos (row 2) to seamless, gravity-aligned 360$^\circ\!$ panoramas. Model inputs and their projected regions are highlighted in red or green. Below each panorama, we show four perspective projections facing left, front, right, and back. Without using explicit camera information, 360Anything handles images with varying Field-of-View and videos with large object and camera motion. The generated consistent panoramas enable 3D scene reconstruction via 3D Gaussian Splatting (row 3). Please see our \Website for results in 360$^\circ\!$ viewers.
• Figure 2: 360Anything pipeline. Given a raw 360$^\circ\!$ training video with arbitrary camera orientations, we first estimate per-frame camera poses and rotate frames to align with the first frame. We then estimate the video's gravity direction and align it with the vertical axis. With such a canonicalized 360$^\circ\!$ video, we project it to a perspective video using randomly sampled camera intrinsics and poses (\ref{['subsec:pano-video-gen']}). We then encode both the conditioning and target videos to latent tokens. Critically, we employ Circular Latent Encoding for the target 360$^\circ\!$ video to avoid seam artifacts in the latent representation. The conditioning tokens (orange) and noisy target tokens (green) are concatenated along the sequence dimension and fed into a diffusion transformer (DiT). The denoised tokens can be decoded back to a 360$^\circ\!$ video via circular latent decoding.
• Figure 3: Illustration of Circular Latent Encoding. The top row (a) shows the seam artifact from naive VAE encoding. Shifting the encoded panorama latent by 180$^\circ\!$ shows a sharp discontinuity at the center, resulting in gray line-like artifacts when decoded back to image. The bottom row (b) illustrates our solution. Before encoding, we apply circular padding to the panorama image. After encoding, the latents in the padded regions are dropped. The shifted latent is now free from discontinuity, providing a seamless latent representation for diffusion training.
• Figure 4: Qualitative results of perspective-to-360$^\circ\!$ image generation. We show multiple perspective views projected from the panorama, where the image with the green border is the conditioning image. Due to the use of a cubemap representation, CubeDiff sometimes generates seams between faces (left). In addition, CubeDiff always assumes the input image has a 90$^\circ\!$ FoV; yet when the actual FoV is smaller, it has to stretch the objects at the image boundary. This leads to distorted object structure, e.g., the balloons (middle) and the mushroom (right). In contrast, 360Anything estimates the correct camera FoV and orientation of the input as shown by the green box on the panorama image, and thus produces much less distorted objects. Please check out our https://360anything.github.io/index.html#360-image-result to view the generated panorama images interactively.
• Figure 5: Qualitative results of perspective-to-360$^\circ\!$ video generation. Regions corresponding to the input conditioning video are highlighted in red. Both Imagine360 and Argus exhibit low visual quality and distortions. ViewPoint always places the conditioning video at the center of the output, and thus generates a rotated image when the video contains large camera motion, leading to distortions (e.g., people and buildings). In contrast, 360Anything generates stably canonicalized panorama videos, and accurately follows the text prompt to outpaint a person holding the camera. Please see our https://360anything.github.io/index.html#compare-with-baseline for better visual comparisons in the video format.