CasaGPT: Cuboid Arrangement and Scene Assembly for Interior Design

TL;DR

CasaGPT tackles interior-scene synthesis by representing objects as assemblies of cuboids and generating their arrangement with a Transformer-based autoregressive model. The approach uses rejection sampling to prune colliding layouts and a refined 3DFRONT-NC dataset to reduce noise and intersections, resulting in more realistic, intersection-free scenes. Extensive experiments on 3D-FRONT and 3DFRONT-NC demonstrate improvements in realism (FID) and collision metrics (CIoU, NIRate) over state-of-the-art methods. This work advances practical 3D scene generation for applications in VR/AR, gaming, and interior design by providing robust, scalable cuboid-based representations and training strategies.

Abstract

We present a novel approach for indoor scene synthesis, which learns to arrange decomposed cuboid primitives to represent 3D objects within a scene. Unlike conventional methods that use bounding boxes to determine the placement and scale of 3D objects, our approach leverages cuboids as a straightforward yet highly effective alternative for modeling objects. This allows for compact scene generation while minimizing object intersections. Our approach, coined CasaGPT for Cuboid Arrangement and Scene Assembly, employs an autoregressive model to sequentially arrange cuboids, producing physically plausible scenes. By applying rejection sampling during the fine-tuning stage to filter out scenes with object collisions, our model further reduces intersections and enhances scene quality. Additionally, we introduce a refined dataset, 3DFRONT-NC, which eliminates significant noise presented in the original dataset, 3D-FRONT. Extensive experiments on the 3D-FRONT dataset as well as our dataset demonstrate that our approach consistently outperforms the state-of-the-art methods, enhancing the realism of generated scenes, and providing a promising direction for 3D scene synthesis.

Figures (21)

• Figure 1: We present CasaGPT, an autoregressive interior design model that frames cuboid primitives decomposed from 3D shapes into an ordered sequence for intersection-free indoor scene generation. We visualize the regional scene arrangement in ❸,❹, and our model is capable of generating compact scene arrangements (❶,❻) despite bounding box intersections (❷,❺).
• Figure 2: Overview of the CasaGPT framework for indoor scene generation. (a) Model Pre-training: Objects in the scene are encoded using distinct tokens representing entities and cuboids. An autoregressive training process in the Transformer Decoder enables the generation of complex layouts. (b) Rejection Sampling: Iterative refinement is applied through cuboid collision inspection to reject unsuitable generation results, followed by model fine-tuning, resulting in high-quality, realistic 3D layouts.
• Figure 3: Workflow of the voxelization, coarse-graining, and cuboid merging process, transforming a 3D object into a compact cuboid representation.
• Figure 4: Comparison of object retrieval methods: (a) Layout model predictions with bounding boxes and cuboids. (b) Nearest neighbor results using bounding boxes, causing intersection issues. (c) Object retrieval using our proposed method prevents intersection, enhancing retrieval accuracy.
• Figure 5: Comparison of the dataset before and after applying intersection avoidance. Our method effectively adjusts object positions to prevent intersections while preserving non-intersecting parts of the scene (third column).