Infinite-World: Scaling Interactive World Models to 1000-Frame Horizons via Pose-Free Hierarchical Memory

TL;DR

Infinite-World tackles long-horizon interactive world modeling in real-world data by addressing pose noise and limited viewpoint revisits. It introduces a Hierarchical Pose-free Memory Compressor to maintain a fixed-budget memory without explicit camera poses, an Uncertainty-aware Action Labeling scheme to robustly map motion to actions, and a Revisit-Dense Finetuning protocol to activate loop-closure with minimal data. The approach yields superior visual quality, memory consistency, and action controllability on open-domain benchmarks and user studies, demonstrating stable 1000-frame horizon performance with bounded computation. This work advances practical real-world neural simulators by enabling long-range memory without relying on perfect pose supervision. It also provides a rigorous evaluation framework combining automated metrics and human judgments to validate long-horizon interactive capabilities.

Abstract

We propose Infinite-World, a robust interactive world model capable of maintaining coherent visual memory over 1000+ frames in complex real-world environments. While existing world models can be efficiently optimized on synthetic data with perfect ground-truth, they lack an effective training paradigm for real-world videos due to noisy pose estimations and the scarcity of viewpoint revisits. To bridge this gap, we first introduce a Hierarchical Pose-free Memory Compressor (HPMC) that recursively distills historical latents into a fixed-budget representation. By jointly optimizing the compressor with the generative backbone, HPMC enables the model to autonomously anchor generations in the distant past with bounded computational cost, eliminating the need for explicit geometric priors. Second, we propose an Uncertainty-aware Action Labeling module that discretizes continuous motion into a tri-state logic. This strategy maximizes the utilization of raw video data while shielding the deterministic action space from being corrupted by noisy trajectories, ensuring robust action-response learning. Furthermore, guided by insights from a pilot toy study, we employ a Revisit-Dense Finetuning Strategy using a compact, 30-minute dataset to efficiently activate the model's long-range loop-closure capabilities. Extensive experiments, including objective metrics and user studies, demonstrate that Infinite-World achieves superior performance in visual quality, action controllability, and spatial consistency.

Figures (8)

• Figure 1: 1000-frame interactive world simulation by Infinite-World. Our model maintains exceptional spatio-temporal consistency and action fidelity over an unprecedented horizon. As the agent explores the indoor environment via keyboard-style controls, Infinite-World accurately renders responsive viewpoint changes and preserves global landmarks (e.g., the window and desk arrangement) even after over 1000 frames, demonstrating robust long-range memory and loop-closure capabilities in complex real-world-style scenarios.
• Figure 2: Overview of Infinite-World architecture.(a) Hierarchical Pose-free Memory Compressor: The Hierarchical Pose-free Memory Compressor (HPMC) recursively compresses raw historical latents into a fixed memory budget via hierarchical compression with local and global stage. The compressor is jointly optimized with the DiT backbone to autonomously anchor generations in the distant past with constant computational cost. (b) Uncertainty-Aware Action labeling: Continuous poses are decoupled into translation and rotation primitives. A tri-state logic filters out "Uncertain" motion to ensure robust action-response learning. (c) Data Strategy: Pre-training on open-domain video is followed by finetuning on a revisit-dense dataset to activate 1000-frame memory consistency.
• Figure 3: Pilot study on spatial memory. Each image is the first frame of a chunk. (Rows 1-3): Memory activation saturates at only 100 training sequences, with loop-closure capability effectively established. (Row 4): Catastrophic collapse occurs during 1000-frame inference as the horizon significantly exceeds the 4-chunk training context.
• Figure 4: Visual comparison between the proposed Infinite-World and four baselines. Notice the visual consistency between 2nd chunk and 6th chunk, and 8th chunk is the zoom-in view of first frame.
• Figure 5: Comparison of memory consumption relative to context video length on an 80GB H800. Our hierarchical compression achieves near-constant memory overhead after an initial growth phase, whereas non-compressed baselines suffer from rapid memory exhaustion.