Tokenize Image as a Set

TL;DR

This work introduces TokenSet, a set-based image representation that tokenizes images into unordered token sets and uses a dual transformation to map sets to fixed-length sequences. A Fixed-Sum Discrete Diffusion framework is proposed to model the resulting structured data under a fixed-sum prior, ensuring a constant token count and discrete representations. Empirical results on ImageNet demonstrate permutation-invariance, enhanced global context, robustness to noise, and semantically coherent token clustering, with reconstruction and generation performance competitive with or superior to serialized-token baselines. The proposed paradigm shifts the design of visual generative models toward permutation-invariant, region-aware coding and constrained discrete diffusion, offering improved robustness and interpretability for token-based image generation.

Abstract

This paper proposes a fundamentally new paradigm for image generation through set-based tokenization and distribution modeling. Unlike conventional methods that serialize images into fixed-position latent codes with a uniform compression ratio, we introduce an unordered token set representation to dynamically allocate coding capacity based on regional semantic complexity. This TokenSet enhances global context aggregation and improves robustness against local perturbations. To address the critical challenge of modeling discrete sets, we devise a dual transformation mechanism that bijectively converts sets into fixed-length integer sequences with summation constraints. Further, we propose Fixed-Sum Discrete Diffusion--the first framework to simultaneously handle discrete values, fixed sequence length, and summation invariance--enabling effective set distribution modeling. Experiments demonstrate our method's superiority in semantic-aware representation and generation quality. Our innovations, spanning novel representation and modeling strategies, advance visual generation beyond traditional sequential token paradigms. Our code and models are publicly available at https://github.com/Gengzigang/TokenSet.

Figures (6)

• Figure 1: Pipeline of our method.
• Figure 2: Pipeline of our set tokenizer.
• Figure 3: Fixed-Sum Diffusion Process. Sample $X_t$ at noise level $t=0.6$ is first sampled from the mixed gaussian distribution of $X_0$ and $X_1$, and then adjusted through greedy adjustment. Samples dropped during greedy adjustment are marked with dashed line.
• Figure 4: Visual comparison of the reconstructed images from various order permutations of the encoded tokens. All reconstructed images are identical, demonstrating the set-based tokenizer is permutation-invariance.
• Figure 5: The receptive fields of individual tokens. Each column represents the receptive field corresponding to the same token. Previous methods such as VQGAN vqgan21patrick encoded tokens strictly correspond to specific positions. In contrast, our approach demonstrates a unique property that many tokens possess global receptive fields.