Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials

TL;DR

This work introduces Make-it-Real, a framework that leverages GPT-4V to recognize and retrieve real-world materials, build a large, richly annotated material library, and automatically generate SVBRDF maps to apply physically plausible materials to 3D objects with only albedo textures. The pipeline comprises rendering-based segmentation, MLLM-driven material matching with hierarchical prompts, and region-to-pixel SVBRDF map synthesis that yields full PBR material sets compatible with standard renderers. Key contributions include (1) the first use of an MLLM for material recognition and assignment in albedo-constrained 3D assets, (2) a 1,400-material library with fine-grained annotations, and (3) a region-to-pixel SVBRDF generation method that maintains albedo consistency while producing complete BRDF maps. The approach significantly improves realism across both CAD-derived and generative-model assets, offering a practical, automated enhancement to the 3D content creation workflow that can be integrated into downstream engines like Blender for photorealistic rendering under dynamic lighting.

Abstract

Physically realistic materials are pivotal in augmenting the realism of 3D assets across various applications and lighting conditions. However, existing 3D assets and generative models often lack authentic material properties. Manual assignment of materials using graphic software is a tedious and time-consuming task. In this paper, we exploit advancements in Multimodal Large Language Models (MLLMs), particularly GPT-4V, to present a novel approach, Make-it-Real: 1) We demonstrate that GPT-4V can effectively recognize and describe materials, allowing the construction of a detailed material library. 2) Utilizing a combination of visual cues and hierarchical text prompts, GPT-4V precisely identifies and aligns materials with the corresponding components of 3D objects. 3) The correctly matched materials are then meticulously applied as reference for the new SVBRDF material generation according to the original albedo map, significantly enhancing their visual authenticity. Make-it-Real offers a streamlined integration into the 3D content creation workflow, showcasing its utility as an essential tool for developers of 3D assets.

Figures (20)

• Figure 1: Usage of Make-it-Real. Our method can refine a wide range of albedo-map-only 3D objects from both CAD design and generative models. Our method enhances the realism of objects, enables part-specific material assignment to objects and generate PBR maps that are compatible with downstream engines.
• Figure 2: Overall pipeline. This pipeline of Make-it-Real is composed of image rendering and material segmentation, MLLM-based material retrieval, and SVBRDF Maps Generation. We finally use blender engine to conduct physically-based rendering.
• Figure 3: The process of MLLM retrieving materials from the Material Library. Utilizing GPT-4V model, we develop a material library, meticulously generating and cataloging comprehensive descriptions for each material. This structured repository facilitates hierarchical querying for material allocation in subsequent looking up processes.
• Figure 4: Illustrations of mask refinement in 2D image space and UV texture space.(a) We effectively cluster concise material-aware masks compared to original segmented parts from li2023semanticsam. (b) We fix missing parts on the uv texture space to get a complete texture partition map.
• Figure 5: Qualitative results of Make-it-Real refining 3D asserts without PBR maps. Objects are selected from Objaverse deitke2022objaverse with albedo maps only.