RenderFlow: Single-Step Neural Rendering via Flow Matching

TL;DR

RenderFlow delivers real-time, photorealistic neural rendering by reframing rendering as a deterministic flow-matching problem that maps G-buffer inputs to shaded images in a single forward pass. It introduces a sparse keyframe guidance mechanism and an adapter-based inverse renderer to boost physical accuracy and enable joint forward/inverse tasks, while leveraging a pretrained video diffusion backbone with albedo conditioning. The approach achieves state-of-the-art fidelity with significantly lower latency than diffusion baselines, demonstrates robust long-sequence performance via progressive inference, and shows versatile applicability to intrinsic decomposition. This work narrows the gap between diffusion-based realism and real-time rendering, with implications for interactive graphics and video editing pipelines.

Abstract

Conventional physically based rendering (PBR) pipelines generate photorealistic images through computationally intensive light transport simulations. Although recent deep learning approaches leverage diffusion model priors with geometry buffers (G-buffers) to produce visually compelling results without explicit scene geometry or light simulation, they remain constrained by two major limitations. First, the iterative nature of the diffusion process introduces substantial latency. Second, the inherent stochasticity of these generative models compromises physical accuracy and temporal consistency. In response to these challenges, we propose a novel, end-to-end, deterministic, single-step neural rendering framework, RenderFlow, built upon a flow matching paradigm. To further strengthen both rendering quality and generalization, we propose an efficient and effective module for sparse keyframe guidance. Our method significantly accelerates the rendering process and, by optionally incorporating sparsely rendered keyframes as guidance, enhances both the physical plausibility and overall visual quality of the output. The resulting pipeline achieves near real-time performance with photorealistic rendering quality, effectively bridging the gap between the efficiency of modern generative models and the precision of traditional physically based rendering. Furthermore, we demonstrate the versatility of our framework by introducing a lightweight, adapter-based module that efficiently repurposes the pretrained forward model for the inverse rendering task of intrinsic decomposition.

Figures (16)

• Figure 1: Overview of the proposed RenderFlow framework.Left:RenderFlow learns a single-step conditional flow from albedo (rather than noise) to shaded images, yielding $\sim$10$\times$ faster rendering while better physical light transport; both forward and inverse rendering are unified within a single model. Right: Example visualizations and corresponding error maps illustrating fidelity.
• Figure 2: An overview of our proposed RenderFlow architecture. Unlike diffusion methods that start from noise, RenderFlow takes albedo as input and directly predicts fully-shaded outputs. Built on a pre-trained video DiT, it injects aligned G-buffer tokens into transformer blocks. Environment maps and optional keyframes are integrated via lightweight adapters (see Fig. \ref{['fig:architecture']} for details). The transformer aggregates all inputs to produce temporally coherent, physically accurate rendering outputs.
• Figure 3: Design of the customized transformer block in RenderFlow. To incorporate global lighting context, the environment map is injected via an adaptive normalization layer before the transformer layers To compensate for the limited information in per-frame G-buffers, the keyframe adapter introduces cross-attention to extract complementary temporal features from an optional keyframe. The adapter includes a LoRA module for parameter-efficient adaptation and employs Rotary Position Embeddings (RoPE) to encode temporal distances between keyframes and current frames.
• Figure 4: Inverse adapter architecture. The inverse adapter uses the frozen forward rendering backbone and introduces specialized modules to adapt it for G-buffer decomposition.
• Figure 5: Qualitative comparison of rendered images with traditional methods including deferred rendering, and diffusion-based approaches (RGB-X and DiffusionRenderer). Our method better preserves texture details and produces high-quality shadows and lighting effects.