L3GO: Language Agents with Chain-of-3D-Thoughts for Generating Unconventional Objects

TL;DR

This work tackles the difficulty diffusion-based models face in obeying precise 3D spatial constraints for unconventional prompts. It introduces L3GO, an LLM-driven agent that builds 3D meshes in Blender via a part-based, feedback-driven loop, using a SimpleBlenv wrapper and a ControlNet-enhanced rendering pathway. The framework is evaluated on ShapeNet-13 and a new UFO benchmark, with human and GPT-4V evaluations showing L3GO outperforms baseline LLM-based approaches in 3D mesh generation and in handling unconventional prompts. The results highlight the potential of integrating language-driven reasoning with 3D modeling workflows to improve controllability and precision, with implications for design, education, and diffusion-model pipelines.

Abstract

Diffusion-based image generation models such as DALL-E 3 and Stable Diffusion-XL demonstrate remarkable capabilities in generating images with realistic and unique compositions. Yet, these models are not robust in precisely reasoning about physical and spatial configurations of objects, especially when instructed with unconventional, thereby out-of-distribution descriptions, such as "a chair with five legs". In this paper, we propose a language agent with chain-of-3D-thoughts (L3GO), an inference-time approach that can reason about part-based 3D mesh generation of unconventional objects that current data-driven diffusion models struggle with. More concretely, we use large language models as agents to compose a desired object via trial-and-error within the 3D simulation environment. To facilitate our investigation, we develop a new benchmark, Unconventionally Feasible Objects (UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender where language agents can build and compose atomic building blocks via API calls. Human and automatic GPT-4V evaluations show that our approach surpasses the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our approach outperforms other state-of-the-art text-to-2D image and text-to-3D models based on human evaluation.

Figures (17)

• Figure 1: We compare one of the state-of-the-art text-to-image models (DALL-E 3) with our LLM-based approach (L3GO). We perform five iterations of DALL-E 3 generation with human feedback but DALL-E 3 does not strictly follow the prompt. L3GO creates a chair with the correct number of legs.
• Figure 2: GPT-4 tries to construct three types of objects from ShapeNet by writing Python scripts in Blender. It can successfully create simple items like lamps, but faces challenges with more complex objects such as tables and airplanes.
• Figure 3: (Top): SimpleBlenv, a wrapper environment on top of Blender, where LLM can construct a 3D mesh by using atomic building blocks. (Bottom): Schematic diagram of L3GO.
• Figure 4: Two types of error feedback we provide in SimpleBlenv: (a) The newly added cuboid (in orange) is completely inside the base cylinder. (b) There is unnecessary spatial gap between the newly added cuboid and the base cylinder.
• Figure 5: GPT-4V evaluation of L3GO, ReAct-B, Reflexion-B, and GPT-4 on ShapeNet-13. 'Human' refers to original ShapeNet meshes that were designed by humans. For complex objects such as airplanes and rifles, L3GO performs better than others.