GO-Renderer: Generative Object Rendering with 3D-aware Controllable Video Diffusion Models

Abstract

Reconstructing a renderable 3D model from images is a useful but challenging task. Recent feedforward 3D reconstruction methods have demonstrated remarkable success in efficiently recovering geometry, but still cannot accurately model the complex appearances of these 3D reconstructed models. Recent diffusion-based generative models can synthesize realistic images or videos of an object using reference images without explicitly modeling its appearance, which provides a promising direction for object rendering, but lacks accurate control over the viewpoints. In this paper, we propose GO-Renderer, a unified framework integrating the reconstructed 3D proxies to guide the video generative models to achieve high-quality object rendering on arbitrary viewpoints under arbitrary lighting conditions. Our method not only enjoys the accurate viewpoint control using the reconstructed 3D proxy but also enables high-quality rendering in different lighting environments using diffusion generative models without explicitly modeling complex materials and lighting. Extensive experiments demonstrate that GO-Renderer achieves state-of-the-art performance across the object rendering tasks, including synthesizing images on new viewpoints, rendering the objects in a novel lighting environment, and inserting an object into an existing video.

Figures (8)

• Figure 1: Results of our GO-Renderer. Driven by multi-view references and 3D poses, GO-Renderer renders multi-view consistent object videos seamlessly integrated into novel environments along arbitrary camera trajectories.
• Figure 2: Overview of GO-Renderer. Given multi-view reference images, we first establish a coarse 3D geometry via 3D reconstruction to render reference and target coordinate maps. These maps bridge explicit 3D geometry with the multi-view reference images via video diffusion model. Specifically, the reference images and their coordinate maps are paired via channel-wise concatenation, while the target coordinate maps are combined with optional appearance guidance. These spatial conditions are then jointly processed by a text-conditioned Video Diffusion Transformer to render high-fidelity object videos with precise viewpoint control.
• Figure 3: Comparision of rendering quality and lighting consistency.
• Figure 4: Qualitative comparison of Multi-view Consistency.
• Figure 5: Quantitative Ablation Results. (a) Impact of the temporal offset $g$ on rendering quality. A sufficient gap ($g=3$) effectively isolates spatial priors. (b) Model robustness under varying degrees of spatial perturbations.