VLMaterial: Procedural Material Generation with Large Vision-Language Models

TL;DR

VLMaterial addresses the challenge of converting a single image into an editable procedural material by fine-tuning a large vision-language model to generate Blender Python code that builds shader node graphs. The authors create an open-source Blender procedural material dataset, and employ data augmentation through a pre-trained LLM to expand the training set, followed by a gradient-free post-optimization stage using MCMC to better match the input appearance. They demonstrate superior perceptual matching over strong baselines on synthetic Blender, synthetic Substance, and real images, and validate practical usability through a user study. The work enables accessible, editable procedural materials and provides a substantial dataset and open-source pipeline for future research in visual program synthesis and inverse procedural graphics.

Abstract

Procedural materials, represented as functional node graphs, are ubiquitous in computer graphics for photorealistic material appearance design. They allow users to perform intuitive and precise editing to achieve desired visual appearances. However, creating a procedural material given an input image requires professional knowledge and significant effort. In this work, we leverage the ability to convert procedural materials into standard Python programs and fine-tune a large pre-trained vision-language model (VLM) to generate such programs from input images. To enable effective fine-tuning, we also contribute an open-source procedural material dataset and propose to perform program-level augmentation by prompting another pre-trained large language model (LLM). Through extensive evaluation, we show that our method outperforms previous methods on both synthetic and real-world examples.

Figures (8)

• Figure 1: Given single input images (left), our model generates corresponding procedural materials (top right) that can be directly applied to a 3D scene (middle). The generated materials can be easily edited, e.g., by changing some node parameters, to achieve desired visual appearance (bottom right).
• Figure 2: Workflow overview. Given an input image, we use our fine-tuned VLM to predict a procedural material in the form of Python code. Executing the code in Blender yields a user-editable shader node graph, whose rendered appearance closely matches the input image.
• Figure 3: A qualitative comparison of our method with several baselines. The input images are shown in the leftmost column. Re-rendered images from our generated procedural material programs best match the input.
• Figure 4: Visual examples for ablation study.
• Figure 5: More qualitative comparisons with baselines.