BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing

TL;DR

BlenderGym introduces a comprehensive, code-editing–based benchmark for evaluating vision-language models on 3D graphics editing across five task categories and 245 start–goal scene pairs. It adopts a generator–verifier pipeline and analyzes how inference scaling, particularly verifier scaling, affects performance, showing that smarter verification can significantly improve edits. The study reveals substantial gaps between state-of-the-art VLMs and human performance, with model- and task-dependent strengths, and demonstrates that allocation of inference compute between generation and verification is a critical lever. BlenderGym offers a fixed, quantitative framework for cross-model comparison and lays groundwork for advancing scalable, automated graphics editing. The findings have practical implications for deploying VLMs in production pipelines and for guiding future research on verification-driven scaling and benchmark design.

Abstract

3D graphics editing is crucial in applications like movie production and game design, yet it remains a time-consuming process that demands highly specialized domain expertise. Automating this process is challenging because graphical editing requires performing a variety of tasks, each requiring distinct skill sets. Recently, vision-language models (VLMs) have emerged as a powerful framework for automating the editing process, but their development and evaluation are bottlenecked by the lack of a comprehensive benchmark that requires human-level perception and presents real-world editing complexity. In this work, we present BlenderGym, the first comprehensive VLM system benchmark for 3D graphics editing. BlenderGym evaluates VLM systems through code-based 3D reconstruction tasks. We evaluate closed- and open-source VLM systems and observe that even the state-of-the-art VLM system struggles with tasks relatively easy for human Blender users. Enabled by BlenderGym, we study how inference scaling techniques impact VLM's performance on graphics editing tasks. Notably, our findings reveal that the verifier used to guide the scaling of generation can itself be improved through inference scaling, complementing recent insights on inference scaling of LLM generation in coding and math tasks. We further show that inference compute is not uniformly effective and can be optimized by strategically distributing it between generation and verification.

Figures (21)

• Figure 1: Example task instances of BlenderGym, a 3D graphics benchmark that tasks VLMs with 3D scene reconstruction through code editing. BlenderGym consists of 245 handcrafted start-goal scene pairs across five key graphics editing tasks: object placement, lighting adjustment, procedural material editing, blend shape manipulation, and procedural geometry editing.
• Figure 2: VLM system setup used by BlenderGym. It follows a generator-verifier structure, where the generator further contains a brainstormer and a code editor. The generator takes in start-goal image pairs along with Python script of the start scene, and then edits the Python script of the start scene to achieve the goal scene based on their visual differences. The verifier takes a pair of renders of proposed edits and selects a single best edit, which is propagated back to the generator for the next iteration.
• Figure 3: Examples of VLM/human generated outputs on lighting and blend shape tasks. Even powerful closed-source VLMs fail to generate lighting settings with accurate colors. In the blend shape task above, only Claude 3.5 Sonnet correctly captures the overall appearance of the police car but still misses the difference of the front light. We report N-CLIP ($10^{-3}$) for all edits above as a calibration.
• Figure 4: Human-VLM and inter-human verifier alignment rate. All models perform above the random baseline (0.5) yet differ notably, with even the highest-aligned model, Claude-3.5-Sonnet (0.66), falling short of inter-human alignment (0.79).
• Figure 5: Verifier scaling results with InternVL2-8B, Claude3.5 Sonnet, and GPT4o. We show that increasing verifier queries brings the selected edit closer to the goal. Bounding boxes of all object instances of interest are shown.