Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BlenderAlchemy: Editing 3D Graphics with Vision-Language Models

Ian Huang, Guandao Yang, Leonidas Guibas

TL;DR

BlenderAlchemy tackles the challenge of editing complex Blender scenes to satisfy user intent expressed in language or reference images by introducing a vision-grounded iterative refinement framework. It represents the Blender state as a base scene plus modular per-task programs and uses a visual edit generator and a visual state evaluator, augmented by a visual-imagination module that grounds prompts with image models. The approach supports procedural material editing, geometry manipulation, and lighting configuration, demonstrating improvements over BlenderGPT and favorable alignment with user prompts through both objective (CLIP) metrics and human judgments. The work highlights practical implications for accelerating 3D design workflows while noting costs, biases, and the need for future improvements in inference speed, domain-specific skill libraries, and end-to-end generation with minimal human in the loop.

Abstract

Graphics design is important for various applications, including movie production and game design. To create a high-quality scene, designers usually need to spend hours in software like Blender, in which they might need to interleave and repeat operations, such as connecting material nodes, hundreds of times. Moreover, slightly different design goals may require completely different sequences, making automation difficult. In this paper, we propose a system that leverages Vision-Language Models (VLMs), like GPT-4V, to intelligently search the design action space to arrive at an answer that can satisfy a user's intent. Specifically, we design a vision-based edit generator and state evaluator to work together to find the correct sequence of actions to achieve the goal. Inspired by the role of visual imagination in the human design process, we supplement the visual reasoning capabilities of VLMs with "imagined" reference images from image-generation models, providing visual grounding of abstract language descriptions. In this paper, we provide empirical evidence suggesting our system can produce simple but tedious Blender editing sequences for tasks such as editing procedural materials and geometry from text and/or reference images, as well as adjusting lighting configurations for product renderings in complex scenes.

BlenderAlchemy: Editing 3D Graphics with Vision-Language Models

TL;DR

BlenderAlchemy tackles the challenge of editing complex Blender scenes to satisfy user intent expressed in language or reference images by introducing a vision-grounded iterative refinement framework. It represents the Blender state as a base scene plus modular per-task programs and uses a visual edit generator and a visual state evaluator, augmented by a visual-imagination module that grounds prompts with image models. The approach supports procedural material editing, geometry manipulation, and lighting configuration, demonstrating improvements over BlenderGPT and favorable alignment with user prompts through both objective (CLIP) metrics and human judgments. The work highlights practical implications for accelerating 3D design workflows while noting costs, biases, and the need for future improvements in inference speed, domain-specific skill libraries, and end-to-end generation with minimal human in the loop.

Abstract

Graphics design is important for various applications, including movie production and game design. To create a high-quality scene, designers usually need to spend hours in software like Blender, in which they might need to interleave and repeat operations, such as connecting material nodes, hundreds of times. Moreover, slightly different design goals may require completely different sequences, making automation difficult. In this paper, we propose a system that leverages Vision-Language Models (VLMs), like GPT-4V, to intelligently search the design action space to arrive at an answer that can satisfy a user's intent. Specifically, we design a vision-based edit generator and state evaluator to work together to find the correct sequence of actions to achieve the goal. Inspired by the role of visual imagination in the human design process, we supplement the visual reasoning capabilities of VLMs with "imagined" reference images from image-generation models, providing visual grounding of abstract language descriptions. In this paper, we provide empirical evidence suggesting our system can produce simple but tedious Blender editing sequences for tasks such as editing procedural materials and geometry from text and/or reference images, as well as adjusting lighting configurations for product renderings in complex scenes.
Paper Structure (40 sections, 1 equation, 29 figures, 3 tables, 2 algorithms)

This paper contains 40 sections, 1 equation, 29 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Task-specific tools for Material Design.
  4. LLM as general problem solvers.
  5. Vision-Language Models.
  6. Method
  7. Representation of the Blender Visual State
  8. Iterative Refinement of Individual Visual Programs
  9. Hypothesis Reversion.
  10. Tweak and Leap Edits.
  11. Visual Imagination.
  12. Experiments
  13. Procedural Material Editing
  14. Text-based material-editing
  15. Image-based material-editing
  16. ...and 25 more sections

Figures (29)

  • Figure 1: Overview of BlenderAlchemy. Given an input Blender state and a user intention specified using either language or reference images, BlenderAlchemy edits the Blender state to satisfy that intention by orange iteratively refining a Blender python program that executes to produce the final Blender state. Our system additionally leverages text-to-image generation for yellow visual imagination, a step that expands a text-only user intention to a concrete visual target to improve program refinement.
  • Figure 2: Iterative visual program editing employs a edit generator $G$ and a state evaluator $V$ in each iteration to explore and prune different potential program edits, where $G$ generates plausible variants of an input program and $V$ picks between two programs based on the consequences they have to the Blender visual state and their alignment to the user intention. Each iteration of the refinement explores variations of the most promising program from the previous iteration. See Algorithm \ref{['alg:iterative_refinement']} for details.
  • Figure 3: Text-based Material Editing Results. The step-by-step edits of a 4x8 version of BlenderAlchemy to the same wooden material, given the text description on the left as the input user intention.
  • Figure 4: The edit discovery process of turning a wooden material into "marbled granite". Each column shows the hypotheses generated by $G$, with the most promising candidates chosen by $V$ indicated by the highlights. Note that iteration 3 proved to be unfruitful according to $V$, and the method reverts to the best candidate from iteration 2, before moving onto iteration 4.
  • Figure 5: Comparisons between our method and BlenderGPT for the text-based material editing task setting. Note how our materials align better with the original language prompts (See Figure \ref{['fig:wood2others']} for the original prompts). The input material being edited is the same wooden material.
  • ...and 24 more figures