Generating Physically Stable and Buildable Brick Structures from Text

TL;DR

BrickGPT reframes brick-assembly design as autoregressive next-brick generation conditioned on text prompts, augmented with physics-aware validity checks. It builds StableText2Brick, a large dataset of stable brick layouts with captions, and tunes an LLM to output brick sequences that are checked for non-collision and static equilibrium. The result is a system that produces diverse, buildable brick structures aligned with prompts and capable of being assembled by humans or robots, with texture/color variants. The work advances design-to-assembly pipelines for physically constrained 3D construction.

Abstract

We introduce BrickGPT, the first approach for generating physically stable interconnecting brick assembly models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of brick structures, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that BrickGPT produces stable, diverse, and aesthetically pleasing brick structures that align closely with the input text prompts. We also develop a text-based brick texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We release our new dataset, StableText2Brick, containing over 47,000 brick structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models at the project website: https://avalovelace1.github.io/BrickGPT/.

Figures (11)

• Figure 1: Quantitative Results. We evaluate our method against several baselines on validity (no out-of-library, out-of-bounds, or colliding bricks), stability, CLIP-based text similarity, and DINOv2-based image similarity. Stability, CLIP, and DINO are computed over valid structures only. For LLaMA-Mesh llamamesh, validity requires a well-formed OBJ file. Results marked "+ our stability analysis" are augmented by generating multiple structures and choosing the first stable one found (if any). Our method outperforms all baselines as well as the ablated setups on validity and stability using our proposed rejection sampling and rollback, while maintaining high text similarity.
• Figure 2: StableText2Brick Dataset. (a) From a ShapeNetCore shapenet2015 mesh, we generate a brick structure by voxelizing it onto a $20\times 20 \times 20$ grid, then constructing its brick layout with a delete-and-rebuild algorithm. (b) We augment each shape with multiple structural variations by randomizing the brick layout while preserving the overall shape. (c) Stability analysis liuStableLegoStabilityAnalysis2024 is performed on each variation to filter out physically unstable designs. (d) To obtain captions for each shape, we render the brick structure from 24 different viewpoints and use GPT-4o achiam2023gpt to generate detailed geometric descriptions. (e) Data samples from 5 categories in our StableText2Brick dataset.
• Figure 3: Method. (a) Our system tokenizes a brick structure into a sequence of text tokens, ordered in a raster-scan manner from bottom to top. (b) We create an instruction dataset pairing brick sequences with descriptions to fine-tune LLaMA-3.2-Instruct-1B. (c) At inference time, BrickGPT generates brick structures incrementally by predicting one brick at a time given a text prompt. For each generated brick, we perform validity checks to ensure it is well-formatted, exists in our brick library, and does not collide with existing bricks. After completing the design, we verify its physical stability. If the structure is unstable, we roll back to a stable state by removing all unstable bricks and their subsequent bricks, and resume generation from that point.
• Figure 4: Force Model. (a) We consider all forces exerted on a single brick, including gravity (black), vertical forces with the top brick (red/blue) and bottom brick (green/purple), and horizontal (shear) forces due to knob connections (cyan), and adjacent bricks (yellow). (b) The structural force model $\mathcal{F}$ extends the individual force model to multiple bricks. Solving for static equilibrium in $\mathcal{F}$ determines each brick's stability score.
• Figure 5: Result Gallery and Baseline Comparisons. Our method generates high-quality, diverse, and novel brick structures aligned with the given text prompts. Black bricks are colliding. For LLaMA-Mesh llamamesh, LGM lgm, XCube XCube, and Hunyuan3D-2 zhao2025hunyuan3d20scalingdiffusion, an inset of the generated mesh is shown in the top-left corner.