RefAny3D: 3D Asset-Referenced Diffusion Models for Image Generation

TL;DR

RefAny3D tackles 3D asset-conditioned image generation by coupling diffusion-based synthesis with 3D geometry priors, introducing a spatially aligned dual-branch architecture that jointly models RGB images and point maps. It employs shared positional embeddings and domain-specific LoRA with text-agnostic attention to enforce cross-domain alignment and prevent background leakage, enabling faithful 3D asset fidelity without per-subject fine-tuning. The method relies on a pose-aligned dataset created from Subjects200k augmented with GroundingDINO, Hunyuan3D, and FoundationPose to provide multi-view RGB–point map conditioning. Experiments show superior 3D-consistency and texture fidelity versus strong baselines across GPT-based and vision-language metrics, suggesting practical potential for integrating 3D assets into diffusion-driven content creation. Overall, RefAny3D broadens diffusion models’ applicability by enabling direct, geometry-aware conditioning on 3D assets for view-consistent 2D renderings.

Abstract

In this paper, we propose a 3D asset-referenced diffusion model for image generation, exploring how to integrate 3D assets into image diffusion models. Existing reference-based image generation methods leverage large-scale pretrained diffusion models and demonstrate strong capability in generating diverse images conditioned on a single reference image. However, these methods are limited to single-image references and cannot leverage 3D assets, constraining their practical versatility. To address this gap, we present a cross-domain diffusion model with dual-branch perception that leverages multi-view RGB images and point maps of 3D assets to jointly model their colors and canonical-space coordinates, achieving precise consistency between generated images and the 3D references. Our spatially aligned dual-branch generation architecture and domain-decoupled generation mechanism ensure the simultaneous generation of two spatially aligned but content-disentangled outputs, RGB images and point maps, linking 2D image attributes with 3D asset attributes. Experiments show that our approach effectively uses 3D assets as references to produce images consistent with the given assets, opening new possibilities for combining diffusion models with 3D content creation.

Figures (13)

• Figure 1: Results of our RefAny3D. Given a 3D asset, our method can generate high-quality and 3D asset-consistent images.
• Figure 2: Overview of RefAny3D. Given a 3D asset, we render multi-view inputs as conditioning signals for the diffusion model and simultaneously generate the point map of the target RGB image. To ensure pixel-level consistency across different viewpoints, we adopt a shared positional encoding strategy. Moreover, to disentangle the RGB domain from the point map domain, we incorporate Domain-specific LoRA and Text-agnostic Attention. Benefiting from this 3D-aware disentanglement design, our method is able to generate high-quality images that maintain strong consistency with the underlying 3D assets.
• Figure 2: Quantitative results of the user study. We evaluate 3D consistency (Faithful), identity preservation (ID), aesthetic quality, and overall ranking (Rank).
• Figure 3: (a) Data construction pipeline. We first use GroundingDINO liu2024grounding to extract the objects of interest, then convert the images into 3D models using Hunyuan3D zhao2025hunyuan3d, and finally apply FoundationPose wen2024foundationpose to estimate the poses of the 3D models in the images. (b) Examples from the dataset.
• Figure 4: Qualitative comparison with other methods. Our approach achieves superior geometric and texture consistency compared to alternative methods.