Procedural Scene Programs for Open-Universe Scene Generation: LLM-Free Error Correction via Program Search

TL;DR

The paper tackles open-universe 3D scene generation by shifting from declarative solvers to an imperative, procedural paradigm. It introduces Procedural Scene Description Language (PSDL), a Python-embedded DSL that encodes explicit object relations, shared parameters, and control-flow, enabling compact, scalable layouts. To address LLM errors without incurring costly calls, the authors propose an LLM-free error-correction mechanism based on iterative local search over program edits, optimizing a loss function that balances layout validity and semantic similarity to the original program. Empirical results from human perceptual studies and a new automated evaluation method show that PS DL with program search outperforms state-of-the-art declarative baselines, especially for large, structured scenes, with competitive runtimes. The work demonstrates practical impact by delivering more robust, flexible scene generation that aligns closely with human judgments while reducing reliance on repeated LLM interactions.

Abstract

Synthesizing 3D scenes from open-vocabulary text descriptions is a challenging, important, and recently-popular application. One of its critical subproblems is layout generation: given a set of objects, lay them out to produce a scene matching the input description. Nearly all recent work adopts a declarative paradigm for this problem: using an LLM to generate a specification of constraints between objects, then solving those constraints to produce the final layout. In contrast, we explore an alternative imperative paradigm, in which an LLM iteratively places objects, with each object's position and orientation computed as a function of previously-placed objects. The imperative approach allows for a simpler scene specification language while also handling a wider variety and larger complexity of scenes. We further improve the robustness of our imperative scheme by developing an error correction mechanism that iteratively improves the scene's validity while staying as close as possible to the original layout generated by the LLM. In forced-choice perceptual studies, participants preferred layouts generated by our imperative approach 82% and 94% of the time when compared against two declarative layout generation methods. We also present a simple, automated evaluation metric for 3D scene layout generation that aligns well with human preferences.

Figures (4)

• Figure 1: Here we show how our error correction mechanism works for the same initial object configuration expressed in 2 languages: Layout-GPT style basic imperative language (left), and PSDL (right). The initial configuration has a positive loss because some objects go out-of-bounds, and some pairs of objects are overlapping. All corrected scenes have loss zero. However, the corrected version expressed in the basic imperative language, and the scene corrected with gradient descent (middle) break the relationship between tables and chairs. Our experiment shows that, given 2 corrections of the same initial object configuration, the automatic evaluator prefers the corrected version expressed in PSDL in 71.4% of cases against LayoutGPT-style, see Table \ref{['tab:ec_ablation']}.
• Figure 2: Human preference study. Top row: per-task vote counts for our method (0/5–5/5) vs DeclBase (left) and Holodeck (right); each sums to 70 tasks. Bottom row: per-participant counts (out of 70) preferred for our method against each baseline.
• Figure 3: Qualitative comparisons between our method, DeclBase, and Holodeck. Our method and Holodeck uses gpt-4o, while DeclBase uses claude-3-5-sonnet-20241022. See the supplemental for a comparison between our method and DeclBase only using claude-3-5-sonnet-20241022.
• Figure 4: More scenes synthesized using our imperative layout generation method with error correction.