Coarse-to-Real: Generative Rendering for Populated Dynamic Scenes

TL;DR

C2R tackles the challenge of generating realistic, controllable videos of populated urban scenes from minimal coarse 3D input. It decouples photorealistic prior learning from structural controllability through a two-stage diffusion-based framework that first learns from real footage and then grounds control using implicit spatio-temporal features, bridged by mixed-domain data. The approach achieves temporally consistent, highly controllable outputs by injecting DINOv3-based control signals and using HSV decorrelation to prevent appearance leakage, with a training mix dominated by real data and a small synthetic anchoring set. In experiments, C2R outperforms baselines in both structural fidelity and visual realism across diverse inputs, demonstrating robust generalization to different engines and cityscapes with practical applicability for content creation and simulation pipelines.

Abstract

Traditional rendering pipelines rely on complex assets, accurate materials and lighting, and substantial computational resources to produce realistic imagery, yet they still face challenges in scalability and realism for populated dynamic scenes. We present C2R (Coarse-to-Real), a generative rendering framework that synthesizes real-style urban crowd videos from coarse 3D simulations. Our approach uses coarse 3D renderings to explicitly control scene layout, camera motion, and human trajectories, while a learned neural renderer generates realistic appearance, lighting, and fine-scale dynamics guided by text prompts. To overcome the lack of paired training data between coarse simulations and real videos, we adopt a two-phase mixed CG-real training strategy that learns a strong generative prior from large-scale real footage and introduces controllability through shared implicit spatio-temporal features across domains. The resulting system supports coarse-to-fine control, generalizes across diverse CG and game inputs, and produces temporally consistent, controllable, and realistic urban scene videos from minimal 3D input. We will release the model and project webpage at https://gonzalognogales.github.io/coarse2real/.

Figures (12)

• Figure 1: Overview of C2R A two-stage framework decouples photorealistic prior learning from structural control: the diffusion backbone is first adapted to real videos, then grounded using implicit spatio-temporal features from mixed real and synthetic data. At inference, a coarse render and text prompt guide denoising to synthesize realistic videos following input motion and layout.
• Figure 2: DINOv3 features provide a domain-robust and temporally stable control signal. We visualize patch embeddings using a global PCA projection computed on a mixed subset of real and synthetic samples, then reused across videos. Similar PCA colors across real and synthetic inputs indicate that DINOv3 aligns corresponding structural elements despite large appearance gaps. Color stability across time suggests temporal coherence even when features are extracted per-frame, supporting spatio-temporal control.
• Figure 3: Dataset samples and domain-bridging strategy.Top: real-world videos used in Stage I to learn a high-fidelity generative prior across diverse cities, clothing styles, weather, and camera motion. Bottom: synthetic HQ/coarse pairs used in Stage II to explicitly teach correspondence between coarse geometry (control) and photorealistic appearance (target).
• Figure 4: Effect of synthetic and real data proportion in training.Top: Training exclusively on synthetic data results in poor alignment with the control input due to lack of variations in data distribution, while training only on real data improves structural alignment but yields weaker contextual richness. Mixing real and synthetic data combines the strengths of both, enabling faithful alignment while encouraging finer detail inpainting. Bottom: We further analyze different synthetic–real data ratios. A small amount of synthetic data (e.g., 1%) enhances detail generation while preserving strong control fidelity. As the proportion of synthetic data increases (5% and 50%), the model becomes progressively more creative but less constrained by the input control signal, leading to increased hallucination and deviation from the intended structure and motion.
• Figure 5: Control signal injection strategies. Direct addition (0 heads), single shared projection (1 head), and per-block projections (10 heads).