Octree Transformer: Autoregressive 3D Shape Generation on Hierarchically Structured Sequences

TL;DR

The paper tackles scalable autoregressive 3D shape generation by combining an octree-based hierarchical representation with a differentiable sequence compression that drastically reduces transformer input length while preserving autoregressiveness. It encodes octrees into structured tokens, compresses them across hierarchy, and decodes via upsampling and masked block convolutions to maintain intra-subtree dependencies. Evaluations on ShapeNet show competitive performance against GAN-based methods at 64^3 resolution and demonstrate voxel superresolution capabilities, while higher resolutions reveal sampling-time and quality challenges. This work advances scalable, autoregressive 3D generation and suggests broader uses for convolution-based compression in long-sequence modeling contexts.

Abstract

Autoregressive models have proven to be very powerful in NLP text generation tasks and lately have gained popularity for image generation as well. However, they have seen limited use for the synthesis of 3D shapes so far. This is mainly due to the lack of a straightforward way to linearize 3D data as well as to scaling problems with the length of the resulting sequences when describing complex shapes. In this work we address both of these problems. We use octrees as a compact hierarchical shape representation that can be sequentialized by traversal ordering. Moreover, we introduce an adaptive compression scheme, that significantly reduces sequence lengths and thus enables their effective generation with a transformer, while still allowing fully autoregressive sampling and parallel training. We demonstrate the performance of our model by comparing against the state-of-the-art in shape generation.

Figures (6)

• Figure 1: The three stages of our network: (a) sequence compression (b) autoregressive generation with transformer (c) sequence decoding
• Figure 2: An example of our octree encoding (for simplicity we depict the encoding for a 2D quadtree). Position values are represented as X/Y for the x- and y-coordinates. Colors are used to indicate the depth level. Filled cells represent full (value=3), white cells empty (value=1) and transparent (value=2) cells mixed voxels. Below are the resulting value and positional sequences.
• Figure 3: By compressing siblings we can achieve compression rates of up to 8 (4 in the quadtree example). For higher compression rates, we consider cousins (of higher order) by compression on the parent level. When compressing parent nodes we replace mixed tokens, by the representation of their children. For tokens that do not have any children (full/empty) we use their original embedding. The result of the compression is a latent vector for each compressed subtree.
• Figure 4: When generating a subtree from a sequence of compressed embeddings, we need to both upsample information from the embedding vectors (black arrows), as well as pass information from all previously generated tokens in the same subtree (green arrows).
• Figure 5: Samples synthesized by the class conditional Octree Transformer at different resolutions.