Cubic Discrete Diffusion: Discrete Visual Generation on High-Dimensional Representation Tokens

Abstract

Visual generation with discrete tokens has gained significant attention as it enables a unified token prediction paradigm shared with language models, promising seamless multimodal architectures. However, current discrete generation methods remain limited to low-dimensional latent tokens (typically 8-32 dims), sacrificing the semantic richness essential for understanding. While high-dimensional pretrained representations (768-1024 dims) could bridge this gap, their discrete generation poses fundamental challenges. In this paper, we present Cubic Discrete Diffusion (CubiD), the first discrete generation model for high-dimensional representations. CubiD performs fine-grained masking throughout the high-dimensional discrete representation -- any dimension at any position can be masked and predicted from partial observations. This enables the model to learn rich correlations both within and across spatial positions, with the number of generation steps fixed at $T$ regardless of feature dimensionality, where $T \ll hwd$. On ImageNet-256, CubiD achieves state-of-the-art discrete generation with strong scaling behavior from 900M to 3.7B parameters. Crucially, we validate that these discretized tokens preserve original representation capabilities, demonstrating that the same discrete tokens can effectively serve both understanding and generation tasks. We hope this work will inspire future research toward unified multimodal architectures. Code is available at: https://github.com/YuqingWang1029/CubiD.

Figures (7)

• Figure 1: Comparison of discrete visual generation approaches. (a) Low-dimensional token generation: Both methods operate at the spatial level—autoregressive requires $h \times w$ sequential steps, while discrete diffusion achieves parallel generation in $T < h \times w$ iterations. (b) High-dimensional token generation: Autoregressive becomes intractable ($h \times w \times d$ steps), and standard discrete diffusion cannot model intra-position dependencies. Our Cubic Discrete Diffusion performs fine-grained masking across the entire 3D tensor—any dimension at any position can be masked and predicted independently—enabling efficient generation in $T \ll h \times w \times d$ iterations while capturing both spatial and dimensional correlations.
• Figure 2: Generated samples from CubiD. Class-conditional generation results on ImageNet 256×256 using high-dimensional representation tokens from DINOv2-B encoder, demonstrating fine details and textures across diverse categories.
• Figure 3: Overview of Cubic Discrete Diffusion. (a) High-dimensional Token Discretization. Given an input image, a frozen representation encoder extracts continuous tokens, which are then discretized through dimension-wise quantization into $h \times w \times d$ discrete tokens. (b) Training via Dimension-wise Mask Modeling. During training, we randomly mask tokens across both spatial and dimensional axes of the tensor (white: masked tokens, pink: visible ground truth tokens, other colors: predicted tokens). The transformer learns to predict these masked tokens from the unmasked context, capturing the complex dependencies across both spatial and dimensional axes.
• Figure 4: Inference process of CubiD. Top row shows the latent token state (white: masked, pink: unmasked), bottom row shows corresponding decoded images. During generation, CubiD starts from a fully masked tensor (0%) and progressively unmasks tokens until reaching a complete image (100%). At each iteration, the model predicts all masked tokens in parallel and randomly unmasks a subset. The percentages show the progress through generation steps. Generation takes hundreds of iterations regardless of feature dimensionality, making high-dimensional discrete generation computationally feasible. The visualization demonstrates a coarse-to-fine generation process, where early iterations establish overall structure and later iterations refine details.
• Figure 5: Qualitative comparison of different masking strategies. Top row: Per-dim masking completely fails, producing severe texture-like artifacts. Middle row: Per-spatial masking generates images with significant local inconsistencies and blurry details. Bottom row: Our per-element masking produces clear, coherent images with fine details. The dramatic quality difference validates that high-dimensional tokens require fine-grained masking across both spatial and dimensional axes.