Scalable Generative Game Engine: Breaking the Resolution Wall via Hardware-Algorithm Co-Design

TL;DR

The paper tackles the memory bandwidth bottleneck limiting real-time, high-resolution neural game engines. It introduces a Hardware-Algorithm Co-Design that decouples the compute-bound World Model from the memory-bound Decoder across a heterogeneous accelerator cluster, augmented by memory-centric operator fusion and manifold-aware latent extrapolation. Key contributions include a theoretical resource-allocation model for DiT/VAE partitioning, hierarchical on-chip fusion to minimize off-chip traffic, and a latent-extrapolation strategy that decouples interaction frequency from generation frequency, achieving real-time performance at $720 \times 480$ with amortized end-to-end latency of $2.7$ ms and up to $26.4$ FPS in continuous and $48.3$ FPS in discrete domains. The results demonstrate that overcoming the Memory Wall via architectural co-design is essential to enable high-fidelity, responsive neural gameplay, broadening the practicality of neural world models for interactive entertainment.

Abstract

Real-time generative game engines represent a paradigm shift in interactive simulation, promising to replace traditional graphics pipelines with neural world models. However, existing approaches are fundamentally constrained by the ``Memory Wall,'' restricting practical deployments to low resolutions (e.g., $64 \times 64$). This paper bridges the gap between generative models and high-resolution neural simulations by introducing a scalable \textit{Hardware-Algorithm Co-Design} framework. We identify that high-resolution generation suffers from a critical resource mismatch: the World Model is compute-bound while the Decoder is memory-bound. To address this, we propose a heterogeneous architecture that intelligently decouples these components across a cluster of AI accelerators. Our system features three core innovations: (1) an asymmetric resource allocation strategy that optimizes throughput under sequence parallelism constraints; (2) a memory-centric operator fusion scheme that minimizes off-chip bandwidth usage; and (3) a manifold-aware latent extrapolation mechanism that exploits temporal redundancy to mask latency. We validate our approach on a cluster of programmable AI accelerators, enabling real-time generation at $720 \times 480$ resolution -- a $50\times$ increase in pixel throughput over prior baselines. Evaluated on both continuous 3D racing and discrete 2D platformer benchmarks, our system delivers fluid 26.4 FPS and 48.3 FPS respectively, with an amortized effective latency of 2.7 ms. This work demonstrates that resolving the ``Memory Wall'' via architectural co-design is not merely an optimization, but a prerequisite for enabling high-fidelity, responsive neural gameplay.

Figures (9)

• Figure 1: Paradigm Shift: From Rendering to Generation. Traditional engines (Top) rely on the CPU to feed explicit geometry to the GPU. Our Heterogeneous Architecture (Bottom) strictly separates the Control Plane (CPU) from the Data Plane (Accelerator Cluster). The World Model (DiT) and Decoders (VAE) communicate exclusively via high-speed interconnects, implementing a zero-copy data flow that overcomes the Memory Wall.
• Figure 2: System Architecture of the Scalable Generative Engine. The architecture strictly separates the Control Plane (CPU) from the Data Plane (Accelerator Cluster). The World Model (DiT) and Decoders (VAE) communicate exclusively via high-speed interconnects, eliminating PCIe bottlenecks.
• Figure 3: Temporal Pipeline Orchestration. By dispatching latent chunks in a Round-Robin fashion to parallel VAE workers, the system hides the physical decoding latency behind the Effective Frame Generation Interval.
• Figure 4: Speculative Control Signal Prefetching. The LSTM predictor anticipates user actions, allowing the system to pre-generate frames and achieve near-zero effective latency on successful predictions.
• Figure 5: Graph Reconstruction for Operator Fusion. The baseline approach (Red) suffers from repeated HBM access for intermediate tensors. Our optimized approach (Green) keeps data within the on-chip SRAM, fusing Upsample, Conv2d, GroupNorm, and SiLU into a single kernel execution. This reduces the effective memory bandwidth requirement by 75%.