Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SceneTeller: Language-to-3D Scene Generation

Başak Melis Öcal, Maxim Tatarchenko, Sezer Karaoglu, Theo Gevers

TL;DR

SceneTeller addresses the barrier to entry in indoor 3D scene design by enabling text-based specification of object layout and appearance. It combines an LLM-driven layout generator, a CAD model retrieval-based scene assembler, and a 3D Gaussian Splatting-based stylization and editing pipeline into a turnkey system. The approach achieves higher geometric fidelity and global consistency over state-of-the-art methods and supports style editing at both scene and object levels. This work lowers the barrier for novices to create realistic 3D rooms for applications such as room planning and game development. The work combines LLM-driven layout reasoning, data-driven model retrieval, and differentiable 3D representations to democratize 3D scene design.

Abstract

Designing high-quality indoor 3D scenes is important in many practical applications, such as room planning or game development. Conventionally, this has been a time-consuming process which requires both artistic skill and familiarity with professional software, making it hardly accessible for layman users. However, recent advances in generative AI have established solid foundation for democratizing 3D design. In this paper, we propose a pioneering approach for text-based 3D room design. Given a prompt in natural language describing the object placement in the room, our method produces a high-quality 3D scene corresponding to it. With an additional text prompt the users can change the appearance of the entire scene or of individual objects in it. Built using in-context learning, CAD model retrieval and 3D-Gaussian-Splatting-based stylization, our turnkey pipeline produces state-of-the-art 3D scenes, while being easy to use even for novices. Our project page is available at https://sceneteller.github.io/.

SceneTeller: Language-to-3D Scene Generation

TL;DR

SceneTeller addresses the barrier to entry in indoor 3D scene design by enabling text-based specification of object layout and appearance. It combines an LLM-driven layout generator, a CAD model retrieval-based scene assembler, and a 3D Gaussian Splatting-based stylization and editing pipeline into a turnkey system. The approach achieves higher geometric fidelity and global consistency over state-of-the-art methods and supports style editing at both scene and object levels. This work lowers the barrier for novices to create realistic 3D rooms for applications such as room planning and game development. The work combines LLM-driven layout reasoning, data-driven model retrieval, and differentiable 3D representations to democratize 3D scene design.

Abstract

Designing high-quality indoor 3D scenes is important in many practical applications, such as room planning or game development. Conventionally, this has been a time-consuming process which requires both artistic skill and familiarity with professional software, making it hardly accessible for layman users. However, recent advances in generative AI have established solid foundation for democratizing 3D design. In this paper, we propose a pioneering approach for text-based 3D room design. Given a prompt in natural language describing the object placement in the room, our method produces a high-quality 3D scene corresponding to it. With an additional text prompt the users can change the appearance of the entire scene or of individual objects in it. Built using in-context learning, CAD model retrieval and 3D-Gaussian-Splatting-based stylization, our turnkey pipeline produces state-of-the-art 3D scenes, while being easy to use even for novices. Our project page is available at https://sceneteller.github.io/.
Paper Structure (31 sections, 5 equations, 3 figures, 5 tables)

This paper contains 31 sections, 5 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. 3D Scene Representations
  4. Text-to-3D Generation
  5. Object-centric Generation.
  6. Scene Generation.
  7. LLMs for Vision
  8. Method
  9. Language-driven 3D Layout Generation
  10. Prompt Construction.
  11. 3D Scene Assembling from Layouts
  12. 3D Scene Stylization
  13. 3D Gaussian Splatting Scene Representation.
  14. Scene Editing.
  15. Experiments
  16. ...and 16 more sections

Figures (3)

  • Figure 1: With solely a textual prompt in natural language describing the desired spatial positions and orientations of objects within the scene, SceneTeller is able to generate realistic and high-quality 3D spaces. Additionally, by facilitating modifications to the style of the entire scene or individual objects within the scene through edit instructions, SceneTeller offers a practical and flexible framework for designing personalized rooms.
  • Figure 2: Overview of our method. Given a textual prompt in natural language delineating the desired spatial positions and orientations of objects within the scene, a 3D scene layout is generated using in-context learning. An initial 3D scene is assembled for the predicted layout, which is then fitted with a 3D Gaussian Splatting representation. This representation is then used to stylize the scene according to the user-provided text prompt, and subsequently render the final images of the scene.
  • Figure 3: Qualitative comparison with state-of-the-art text-to-3D scene generation methods. The top/upper walls are marked with arrows for your reference, if this information is available within the generations. SceneTeller is able to generate high-quality scenes, with superior geometric fidelity and 3D consistency.