Re-Thinking Inverse Graphics With Large Language Models

TL;DR

Inverse graphics aims to reconstruct a 3D scene from a single image, but generalization across domains remains difficult. The authors propose IG-LLM, an inverse-graphics framework that fine-tunes an instruction-tuned LLM fed by a frozen CLIP visual encoder to output a structured graphics program, with a numeric head enabling continuous metric reasoning and a training objective $p(x)=\prod_{i=1}^n p(s_i|s_1,...,s_{i-1})$. They demonstrate strong compositional generalization, parameter-space generalization across 2D and SO(3) spaces, and 6-DoF pose estimation on synthetic data, with the numeric head yielding improved precision and data efficiency. The work shows that broad world knowledge encoded in LLMs can be repurposed for inverse graphics, offering a data-efficient path toward holistic scene understanding and highlighting future directions such as differentiable rendering and real-world generalization.

Abstract

Inverse graphics -- the task of inverting an image into physical variables that, when rendered, enable reproduction of the observed scene -- is a fundamental challenge in computer vision and graphics. Successfully disentangling an image into its constituent elements, such as the shape, color, and material properties of the objects of the 3D scene that produced it, requires a comprehensive understanding of the environment. This complexity limits the ability of existing carefully engineered approaches to generalize across domains. Inspired by the zero-shot ability of large language models (LLMs) to generalize to novel contexts, we investigate the possibility of leveraging the broad world knowledge encoded in such models to solve inverse-graphics problems. To this end, we propose the Inverse-Graphics Large Language Model (IG-LLM), an inverse-graphics framework centered around an LLM, that autoregressively decodes a visual embedding into a structured, compositional 3D-scene representation. We incorporate a frozen pre-trained visual encoder and a continuous numeric head to enable end-to-end training. Through our investigation, we demonstrate the potential of LLMs to facilitate inverse graphics through next-token prediction, without the application of image-space supervision. Our analysis enables new possibilities for precise spatial reasoning about images that exploit the visual knowledge of LLMs. We release our code and data at https://ig-llm.is.tue.mpg.de/ to ensure the reproducibility of our investigation and to facilitate future research.

Figures (18)

• Figure 1: IG-LLM. We present the Inverse-Graphics Large Language Model (IG-LLM) framework, a general approach to solving inverse-graphics problems. We instruction-tune an LLM to decode a visual (CLIP) embedding into graphics code that can be used to reproduce the observed scene using a standard graphics engine. Leveraging the broad reasoning abilities of LLMs, we demonstrate that our framework exhibits natural generalization across a variety of distribution shifts without the use of special inductive biases.
• Figure 2: Numeric Head. (\ref{['ssec:numeric_head']}) Rather than producing digits as discrete tokens (a), we train our model to generate a [NUM] token when a number should be produced. The [NUM] token is used as a mask to signal the embedding should instead be passed through the numeric head, preserving the gradient (b).
• Figure 3: OOD CLEVR-CoGenT Samples. (\ref{['ssec:clevr']}) NS-VQA, with its modular design, fails to disentangle shape from color, while our framework is able to effectively generalize to OOD attribute combinations. See \ref{['fig:clevr_samples_additional']} for additional samples.
• Figure 4: CLEVR Data Efficiency. (\ref{['ssec:clevr']}) Plot of the validation L2 positional error by the number of training samples. We observe that the float-based model is consistently more data-efficient but that the difference between the models converges as the number of training samples reaches 4000. See \ref{['table:clevr_data_efficiency']} for a full quantitative comparison.
• Figure 5: 2D Parameter-Space Generalization. (\ref{['sssection:2d']}) (a) Training positions are sampled from the checkerboard. When evaluated on images with uniformly sampled positions, the char-based model fails to generalize outside the training distribution (b) while the float-based model effectively interpolates samples (c). Randomly sampled testing locations are shown in red and the corresponding predictions in blue. (d) shows that, while both methods well estimate samples from the ID condition, the char-based model struggles to generalize. (e) shows a plot of the model's validation MSE as a function of the number of training steps. We observe that the training of the float-based model is much smoother and converges quickly.