The Scene Language: Representing Scenes with Programs, Words, and Embeddings

TL;DR

The Scene Language addresses the challenge of rich visual scene representation by unifying structure, semantics, and identity as a triplet $Φ(s) = (W, P, Z)$ that is executable via a DSL of entity functions. A training-free inference pipeline uses pre-trained language models to map text or images to $W$, $P$, and $Z$, enabling rendering across diverse engines and enabling precise editing through interpretable programs. The framework supports 3D and 4D scene generation and editing, with empirical evidence showing improved prompt alignment and editing fidelity over scene-graph and latent-space baselines, and cross-renderer consistency. By decoupling structure (programs) from semantics (words) and appearance (embeddings), the approach delivers controllable, scalable, and high-fidelity scene synthesis suitable for applications in 3D/4D content creation. $Φ(s) = (W, P, Z)$ provides a compact formalism that underpins the end-to-end pipeline from LM-based inference to renderer-based image synthesis.

Abstract

We introduce the Scene Language, a visual scene representation that concisely and precisely describes the structure, semantics, and identity of visual scenes. It represents a scene with three key components: a program that specifies the hierarchical and relational structure of entities in the scene, words in natural language that summarize the semantic class of each entity, and embeddings that capture the visual identity of each entity. This representation can be inferred from pre-trained language models via a training-free inference technique, given text or image inputs. The resulting scene can be rendered into images using traditional, neural, or hybrid graphics renderers. Together, this forms a robust, automated system for high-quality 3D and 4D scene generation. Compared with existing representations like scene graphs, our proposed Scene Language generates complex scenes with higher fidelity, while explicitly modeling the scene structures to enable precise control and editing.

Figures (15)

• Figure 1: Structured Scene Generation and Editing Using the Scene Language. We develop a scene representation for 3D scene generation and editing tasks. Given textual scene descriptions, the representation can be inferred by a pre-trained large language model, rendered in 3D, and edited following language instructions. The representation contains a program consisting of semantic-aware functions bound to words, providing high interpretability and an intuitive scene-editing interface, and embeddings enabling editing with fine controls, e.g., transferring the style of <z1*> from a user's image to the output by updating <z1> which controls global attributes of the scene. Project page: https://ai.stanford.edu/ yzzhang/projects/scene-language/.
• Figure 2: Overview. A Scene Language represents a scene with three components: a program consisting of entity functions, a set of words (e.g., pawn) denoting the semantic class of the entity functions, and a list of embeddings (e.g., <z1>) capturing the identity of each entity in the scene. Each entity function is bound with an entity class name given by a word, and maps an input embedding to an output entity of that class. Executing the program evaluates entity functions to compute the full set of entities in the scene. The resulting computation graph, illustrated on the right, captures the dependency structure among entities (indicated by arrows). The program shown is converted from our text-conditioned inference method's output, with details included in \ref{['app:lm-output-text-cond']}; it is written in Lisp-style syntax for brevity and is implemented with Python in practice (\ref{['ssec:dsl']}).
• Figure 3: Rendering. Given a Scene Language in (a), a program interpreter executes the program to obtain a data object in (b). A graphics renderer first reparameterizes the data object from (b) into the renderer-specific parameter space, and then executes the rendering operation $\mathcal{R}$ to obtain the final image output in (d).
• Figure 4: Text-Prompted Scene Generation. Ours produce more accurate outputs than using no intermediate representation (MVDream) or scene graph (GraphDreamer).
• Figure 5: Renderings Across Graphics Renderers. Different renderers produce renderings that adhere to the same representation and are therefore visually aligned, while each exhibits a different imaging style. Text inputs are shown at the subfigure bottoms.