From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries

TL;DR

FactoredScenes tackles real-world indoor scene generation under limited data by factorizing scenes into room-layout programs and pose variations. It learns a library of reusable room-structure functions from synthetic data, uses an LLM to generate scene programs regularized by this library, executes the programs to obtain layouts, and then hierarchically predicts oriented object poses before retrieving concrete 3D instances. The approach yields state-of-the-art realism on ScanNet-like data, with strong quantitative metrics (FID/KID) and human studies showing generated rooms closely resemble real ones, even when tested on unseen scenes. This modular, library-regularized pipeline enables leveraging diverse data sources and provides a scalable path toward realistic, program-driven scene synthesis for datasets and benchmarks.

Abstract

Real-world scenes, such as those in ScanNet, are difficult to capture, with highly limited data available. Generating realistic scenes with varied object poses remains an open and challenging task. In this work, we propose FactoredScenes, a framework that synthesizes realistic 3D scenes by leveraging the underlying structure of rooms while learning the variation of object poses from lived-in scenes. We introduce a factored representation that decomposes scenes into hierarchically organized concepts of room programs and object poses. To encode structure, FactoredScenes learns a library of functions capturing reusable layout patterns from which scenes are drawn, then uses large language models to generate high-level programs, regularized by the learned library. To represent scene variations, FactoredScenes learns a program-conditioned model to hierarchically predict object poses, and retrieves and places 3D objects in a scene. We show that FactoredScenes generates realistic, real-world rooms that are difficult to distinguish from real ScanNet scenes.

Figures (17)

• Figure 1: We propose FactoredScenes, a framework that generates layout programs which encode underlying room structure, and predicts objects poses that capture nuanced variation in real-world, lived-in scenes.
• Figure 2: The FactoredScenes framework. FactoredScenes (i) learns a library of programs, (ii) generates a scene program with the learned library, (iii) executes the program to retrieve layouts, (iv) predicts object poses given the program, and (v) retrieves object instances for the full scene. This factorization enables our framework to use different sources of data to generate real-world scenes.
• Figure 3: FactoredScenes employs an alternating wake-sleep formulation; in the wake stage, an LLM generates the underlying program of a layout, and in the sleep stage, an LLM proposes new abstractions for the library given successful reconstructions.
• Figure 4: FactoredScenes hierarchically predicts poses for objects based on the underlying program. Our model first predicts primary objects' poses with an attention-based model (e.g., table), then predicts dependent objects' poses additionally conditioned on the orientation and program of their dependency target (e.g., the chairs' poses are dependent on that of table's).
• Figure 5: Our predicted layout better matches that of ScanNet scenes compared to prior works.