I3DM: Implicit 3D-aware Memory Retrieval and Injection for Consistent Video Scene Generation

Abstract

Despite remarkable progress in video generation, maintaining long-term scene consistency upon revisiting previously explored areas remains challenging. Existing solutions rely either on explicitly constructing 3D geometry, which suffers from error accumulation and scale ambiguity, or on naive camera Field-of-View (FoV) retrieval, which typically fails under complex occlusions. To overcome these limitations, we propose I3DM, a novel implicit 3D-aware memory mechanism for consistent video scene generation that bypasses explicit 3D reconstruction. At the core of our approach is a 3D-aware memory retrieval strategy, which leverages the intermediate features of a pre-trained Feed-Forward Novel View Synthesis (FF-NVS) model to score view relevance, enabling robust retrieval even in highly occluded scenarios. Furthermore, to fully utilize the retrieved historical frames, we introduce a 3D-aligned memory injection module. This module implicitly warps historical content to the target view and adaptively conditions the generation on reliable warping regions, leading to improved revisit consistency and accurate camera control. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches, achieving superior revisit consistency, generation fidelity, and camera control precision.

Figures (7)

• Figure 1: Overview of I3DM, an implicit 3D-aware memory mechanism for consistent video generation. Given an input image and a user-specified camera trajectory, I3DM enables consistent scene exploration via a 3D-aware memory retrieval network and a 3D-aligned memory injection module. Our method ensures consistent revisiting (indicated by frames with matching colors), even under complex occlusions.
• Figure 2: Limitations of existing memory mechanisms. (Top-Left) Explicit geometry-based methods (e.g., Gen3C ren2025gen3c) suffer from scale estimation ambiguity, leading to inaccurate camera navigation (e.g., colliding with the wall) and revisit inconsistencies. (Top-Right) Implicit FoV-based methods (e.g., WorldPlay sun2025worldplay) fail under occlusions, as FoV overlap ignores actual visual visibility. This retrieves irrelevant historical frames, causing repeated semantic content and inconsistent revisits. (Bottom) Visual examples. Frames with matching colors denote the same viewpoint and should be strictly consistent. Red circles highlight inconsistencies, and red box indicates the repeated content.
• Figure 3: Overview of the proposed I3DM framework. Left: 3D-aware Memory Retrieval. For each historical frame in the memory bank, we first extract 3D-aware intermediate features using a pre-trained NVS model. A lightweight scoring network then evaluates their spatial relevance to the target view to select the most relevant frames. Right: 3D-aligned Memory Injection. The retrieved frames are processed by an Adaptive NVS Module to align them with the target view. These aligned results are then used to condition the Wan-DiT backbone for consistent video scene generation.
• Figure 4: 3D-aware Memory Retrieval Module. We first set the last frame as an anchor. For each historical candidate frame, we extract implicit 3D-aware features from the $l^{\text{th}}$ layer of the frozen LVSM. Based on these features, a Scoring CNN then predicts a spatial uncertainty map for the target view. Notably, the full Transformer is executed only during training to provide supervision; during inference, the process terminates at the $l^{\text{th}}$ layer to efficiently deduce the score map.
• Figure 5: Qualitative comparison on the Re10K (top) and T&T (bottom) datasets. Black dashed arrows link corresponding frames that should remain consistent; red circles and arrows highlight visual inconsistencies and inaccurate camera motion, respectively.