NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos

TL;DR

NeoVerse presents a scalable 4D world model that learns from in-the-wild monocular videos to achieve state-of-the-art 4D reconstruction and controllable novel-trajectory video generation. It keyly introduces pose-free feed-forward 4D Gaussian Splatting (4DGS) with bidirectional motion modeling, plus online monocular degradation simulation to train generation under degraded conditions. The framework couples reconstruction and generation in a two-stage training process, enabling efficient on-the-fly reconstruction from sparse keyframes and degradation-conditioned generation, validated by comprehensive static/dynamic reconstruction benchmarks and generation tasks. NeoVerse demonstrates strong performance across 3D tracking, video editing, stabilization, and super-resolution, highlighting its potential for versatile, scalable 4D world modeling from widely available monocular video data.

Abstract

In this paper, we propose NeoVerse, a versatile 4D world model that is capable of 4D reconstruction, novel-trajectory video generation, and rich downstream applications. We first identify a common limitation of scalability in current 4D world modeling methods, caused either by expensive and specialized multi-view 4D data or by cumbersome training pre-processing. In contrast, our NeoVerse is built upon a core philosophy that makes the full pipeline scalable to diverse in-the-wild monocular videos. Specifically, NeoVerse features pose-free feed-forward 4D reconstruction, online monocular degradation pattern simulation, and other well-aligned techniques. These designs empower NeoVerse with versatility and generalization to various domains. Meanwhile, NeoVerse achieves state-of-the-art performance in standard reconstruction and generation benchmarks. Our project page is available at https://neoverse-4d.github.io

Figures (13)

• Figure 1: Illustration of NeoVerse. NeoVerse reconstructs 4D Gaussian Splatting (4DGS) from monocular videos in a feed-forward manner. These 4DGS can be rendered from novel viewpoints to provide degraded rendering conditions for generating high-quality and spatial-temporally coherent videos.
• Figure 2: Framework of NeoVerse. In the reconstruction part, we propose a pose-free feed-forward 4DGS reconstruction model (\ref{['sec:recon']}) with bidirectional motion modeling. The degraded renderings in novel viewpoints from 4DGS are input to the generation model as conditions. During training, the degraded rendering conditions are simulated from monocular videos (\ref{['sec:gen']}), and the original videos themselves serve as targets.
• Figure 3: Training pairs with degradation simulation.
• Figure 4: Generation with large camera motions on challenging in-the-wild videos. We compare our method against other related work on "Pan left" (left) and "Move right" (right) cases. Our NeoVerse achieves better generation quality while maintaining precise camera controllability. Yellow boxes highlight artifacts.
• Figure 5: Qualitative comparison with state-of-the-art methods in static scenes. Red boundaries indicate inconsistent renderings due to inaccurate pose prediction. Yellow boxes indicate artifacts.