FB-4D: Spatial-Temporal Coherent Dynamic 3D Content Generation with Feature Banks

TL;DR

FB-4D addresses the challenge of high-fidelity, temporally coherent dynamic 4D content generation without requiring training or fine-tuning. It introduces a feature bank that stores compact representations from past frames and fuses them into current-frame generation via enhanced self-attention, with a dynamic merging mechanism to keep the bank small yet representative. By coupling this feature bank with autoregressive, multi-iteration generation and progressive viewpoint selection, FB-4D yields consistent multi-view sequences and a final deformable 3D Gaussian representation for 4D content. Experimental results on Consistent4D demonstrate state-of-the-art performance for training-free methods and competitiveness with training-based SV4D, validating the effectiveness of implicit cross-frame correspondences learned through the feature bank. The approach offers a practical, memory-efficient path toward robust 4D content generation suitable for real-world applications.

Abstract

With the rapid advancements in diffusion models and 3D generation techniques, dynamic 3D content generation has become a crucial research area. However, achieving high-fidelity 4D (dynamic 3D) generation with strong spatial-temporal consistency remains a challenging task. Inspired by recent findings that pretrained diffusion features capture rich correspondences, we propose FB-4D, a novel 4D generation framework that integrates a Feature Bank mechanism to enhance both spatial and temporal consistency in generated frames. In FB-4D, we store features extracted from previous frames and fuse them into the process of generating subsequent frames, ensuring consistent characteristics across both time and multiple views. To ensure a compact representation, the Feature Bank is updated by a proposed dynamic merging mechanism. Leveraging this Feature Bank, we demonstrate for the first time that generating additional reference sequences through multiple autoregressive iterations can continuously improve generation performance. Experimental results show that FB-4D significantly outperforms existing methods in terms of rendering quality, spatial-temporal consistency, and robustness. It surpasses all multi-view generation tuning-free approaches by a large margin and achieves performance on par with training-based methods.

Figures (14)

• Figure 1: Compared to STAG4D zeng2025stag4d, our method enhances spatial consistency (top row, better texture alignment across different viewpoints) and temporal consistency (bottom row, with fewer floaters and smoother motion transitions) by leveraging implicit correspondences in diffusion features. This translates to a superior FVD score than STAG4D (724.26 v.s. 992.21).
• Figure 2: Paradigm comparison of our method with previous works. (a) Zero123++ uses a dual-branch self-attention mechanism, one branch for the reference image and the other for noised latent, with the reference image enhancing spatial consistency. (b) STAG4D adds key-value information from the first frame to improve frame consistency. (c) Our method introduces a feature bank that stores a compact representation deduced from the previous $i-1$ frames, with the same size as a single frame but contains richer feature temporally. This bank is used to generate $F^{[v=0,t=i]}$. After obtaining the output $\textbf{O}_\textbf{i}$ from the self-attention block, we blend it with the stored $\textbf{O}_\textbf{i}^{\text{fb}}$ from the feature bank to enhance temporal consistency (detailed in Sec. \ref{['sec32']}).
• Figure 3: Overall pipeline of video-to-4D generation in FB-4D. Given a single-view video input, we integrate a multi-view generative diffusion model shi2023zero123++ with a feature bank to enhance spatial and temporal consistency, implicitly modeling the correspondence across views and timestamps (detailed in Sec. \ref{['sec32']}). We perform multiple iterations of generation, where each generated view is used as input for the next iteration. For example, in the first iteration, we use $F^{[v=0,t=1:T]}$ to generate multi-view sequences $F^{[v=1:6,t=1:T]}$ and in the second iteration, we select the input view progressively (detailed in Sec. \ref{['sec33']}). The input for this iteration becomes $F^{[v=k_1,t=1:T]}$, where $k_1 \in [1,6]$. This iterative process enables interaction across feature banks, refining the multi-view sequences over time. The generated sequences maintain high consistency in both spatial and temporal dimensions. After several iterations, we finally train a deformable 3D Gaussian to represent the 4D model.
• Figure 4: Illustration of the compact feature fusion in the updating process. The tokens in the feature bank are continuously updated through the aforementioned four steps, effectively integrating past features while ensuring that the size of the feature bank remains constant.
• Figure 5: Illustration of feature bank fetching and updating. In the multi-view inference process of frame $F^{[v=0, t=i]}$, we first retrieve the corresponding feature set for computation and then utilize the keys, values, and outputs ($\mathbf{\textbf{K}}_\textbf{i}$, $\mathbf{\textbf{V}}_\textbf{i}$, $\mathbf{\textbf{O}}_\textbf{i}$) from the self-attention blocks to update the feature bank.