MaterialFusion: High-Quality, Zero-Shot, and Controllable Material Transfer with Diffusion Models

TL;DR

The paper tackles exemplar-based material transfer in natural images, aiming for high fidelity transfer with preserved geometry and backgrounds in a zero-shot setting. It introduces MaterialFusion, a unified diffusion-based framework that fuses GaR (Guide-and-Rescale) with IP-Adapter, supplemented by a dual masking strategy and a Material Transfer Force parameter $\lambda$. The method leverages Stable Diffusion v1.5, classifier-free guidance, and DDIM inversion to achieve controlled edits, and demonstrates superior quality and balance between transfer and detail preservation over baselines, supported by quantitative metrics (CLIP, LPIPS) and user studies. Code is released, enabling replication and application to real-world editing and visualization workflows.

Abstract

Manipulating the material appearance of objects in images is critical for applications like augmented reality, virtual prototyping, and digital content creation. We present MaterialFusion, a novel framework for high-quality material transfer that allows users to adjust the degree of material application, achieving an optimal balance between new material properties and the object's original features. MaterialFusion seamlessly integrates the modified object into the scene by maintaining background consistency and mitigating boundary artifacts. To thoroughly evaluate our approach, we have compiled a dataset of real-world material transfer examples and conducted complex comparative analyses. Through comprehensive quantitative evaluations and user studies, we demonstrate that MaterialFusion significantly outperforms existing methods in terms of quality, user control, and background preservation. Code is available at https://github.com/ControlGenAI/MaterialFusion.

Figures (14)

• Figure 1: Overview of the material transfer process in MaterialFusion. Starting with a material exemplar $y_{im}$, an input image $x_{init}$, and prompts, our framework produces a target image where the object adopts the desired material properties from $y_{im}$.
• Figure 2: (Left) Comparison of material transfer results across different methods. From left to right: the original image, target material, results using Guide-and-Rescale (GaR), IP-Adapter with masking, our method without masking, and our full MaterialFusion approach. Our method achieves realistic material transfer while preserving object structure and background consistency. (Right) Gradual transfer of material characteristics with increasing "transfer force" ($\lambda$).
• Figure 3: First Masking. After the first masking, the material is successfully transferred to the targeted area of the image. However, background preservation is not flawless, with noticeable issues occurring on the table.
• Figure 4: The overall pipeline of MaterialFusion for material transfer. Starting with DDIM inversion of the target image $x_{init}$ and material exemplar $y_{im}$, the framework combines the IP-Adapter with UNet and employs a guider energy function for precise material transfer. A dual-masking strategy ensures material application only on target regions while preserving background consistency, ultimately generating the edited output $x_{edit}$. The parameter $\lambda$, known as the Material Transfer Force, controls the intensity of the material application, enabling adjustment of the transfer effect according to user preference.
• Figure 5: To compare the qualitative results obtained by different methods: Our method, ZeST, GaR, and IP-Adapter with masking. Our method demonstrates more realistic material integration, preserving object structure and achieving higher fidelity to the target material.