Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Composable Visual Tokenizers with Generator-Free Diagnostics of Learnability

Bingchen Zhao, Qiushan Guo, Ye Wang, Yixuan Huang, Zhonghua Zhai, Yu Tian

TL;DR

This work tackles the challenge of learning visual tokenizers that are both information-rich and easily learnable by downstream generators. It introduces CompTok, a tokenizer framework using a token-conditioned diffusion decoder, augmented by an InfoGAN-style mutual-information objective, token swapping for compositionality, and an Adversarial Flow Model to keep swapped tokens on the natural-image manifold. To assess tokenizer usefulness beyond reconstruction, the authors propose generator-free diagnostics AvgIG and MC that predict downstream learnability and task utility, and show these metrics correlate with downstream performance better than traditional rFID. Empirically, CompTok achieves state-of-the-art class-conditioned generation, enables high-level semantic editing via token swaps, and demonstrates robust correlations with task utility across remote sensing, medical imaging, and OCR domains.

Abstract

We introduce CompTok, a training framework for learning visual tokenizers whose tokens are enhanced for compositionality. CompTok uses a token-conditioned diffusion decoder. By employing an InfoGAN-style objective, where we train a recognition model to predict the tokens used to condition the diffusion decoder using the decoded images, we enforce the decoder to not ignore any of the tokens. To promote compositional control, besides the original images, CompTok also trains on tokens formed by swapping token subsets between images, enabling more compositional control of the token over the decoder. As the swapped tokens between images do not have ground truth image targets, we apply a manifold constraint via an adversarial flow regularizer to keep unpaired swap generations on the natural-image distribution. The resulting tokenizer not only achieves state-of-the-art performance on image class-conditioned generation, but also demonstrates properties such as swapping tokens between images to achieve high level semantic editing of an image. Additionally, we propose two metrics that measures the landscape of the token space that can be useful to describe not only the compositionality of the tokens, but also how easy to learn the landscape is for a generator to be trained on this space. We show in experiments that CompTok can improve on both of the metrics as well as supporting state-of-the-art generators for class conditioned generation.

Composable Visual Tokenizers with Generator-Free Diagnostics of Learnability

TL;DR

This work tackles the challenge of learning visual tokenizers that are both information-rich and easily learnable by downstream generators. It introduces CompTok, a tokenizer framework using a token-conditioned diffusion decoder, augmented by an InfoGAN-style mutual-information objective, token swapping for compositionality, and an Adversarial Flow Model to keep swapped tokens on the natural-image manifold. To assess tokenizer usefulness beyond reconstruction, the authors propose generator-free diagnostics AvgIG and MC that predict downstream learnability and task utility, and show these metrics correlate with downstream performance better than traditional rFID. Empirically, CompTok achieves state-of-the-art class-conditioned generation, enables high-level semantic editing via token swaps, and demonstrates robust correlations with task utility across remote sensing, medical imaging, and OCR domains.

Abstract

We introduce CompTok, a training framework for learning visual tokenizers whose tokens are enhanced for compositionality. CompTok uses a token-conditioned diffusion decoder. By employing an InfoGAN-style objective, where we train a recognition model to predict the tokens used to condition the diffusion decoder using the decoded images, we enforce the decoder to not ignore any of the tokens. To promote compositional control, besides the original images, CompTok also trains on tokens formed by swapping token subsets between images, enabling more compositional control of the token over the decoder. As the swapped tokens between images do not have ground truth image targets, we apply a manifold constraint via an adversarial flow regularizer to keep unpaired swap generations on the natural-image distribution. The resulting tokenizer not only achieves state-of-the-art performance on image class-conditioned generation, but also demonstrates properties such as swapping tokens between images to achieve high level semantic editing of an image. Additionally, we propose two metrics that measures the landscape of the token space that can be useful to describe not only the compositionality of the tokens, but also how easy to learn the landscape is for a generator to be trained on this space. We show in experiments that CompTok can improve on both of the metrics as well as supporting state-of-the-art generators for class conditioned generation.
Paper Structure (36 sections, 5 theorems, 21 equations, 8 figures, 3 tables)

This paper contains 36 sections, 5 theorems, 21 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem Setting
  5. Why rFID Can Mislead on gFID
  6. Metrics: AvgIG and MC
  7. Diffusion Decoder D
  8. Aligning Tokenizer Training with Completeness and Learnability
  9. Mutual-Information Token Supervision
  10. Swap-Regularized Compositionality
  11. Manifold Constraint via Adversarial Flow Model (AFM)
  12. Downstream Training of $G$: MaskGIT with a Diffusion Token Head
  13. MaskGIT backbone.
  14. Continuous tokens: diffusion regression head.
  15. Experiments
  16. ...and 21 more sections

Key Result

Lemma C.3

Assume a Gaussian probabilistic decoder $p(x|z) = \mathcal{N}(x; D(z), \sigma^2 I)$. Then, the term $\Delta I_t(x)$ is exactly equal to the increase in log-likelihood of the target $x$ achieved by the update step $z_t \to z_{t+1}$ (shifted by a constant).

Figures (8)

  • Figure 1: CompTok overview. Left: a reconstruction pathway where the encoder $E$ produces a 1D token sequence that is decoded by $D$; a recognition head $Q_\phi$ must recover the tokens from the decoded image (mutual-information loss $\mathcal{L}_{\mathrm{MI}}$), while an adversarial flow/density model $\psi$ enforces realism via $\mathcal{L}_{\mathrm{AFM}}$ alongside the reconstruction loss $\mathcal{L}_{\mathrm{tok}}$. Right: a swap pathway where tokens from two images are partially exchanged, the mixed token is decoded, and the same recoverability and realism constraints $\mathcal{L}_{\mathrm{MI}}^{\mathrm{swap}}$ and $\mathcal{L}_{\mathrm{AFM}}$ train tokens to be non-degenerate and compositional under edits.
  • Figure 2: AvgIG and MC are computed on the tokenizer (E,D) to evaluate the quality of the tokenizer and the downstream generator. AvgIG measures the average information gain per optimization step when fitting a latent token to a target image, and MC measures the pairwise mode connectivity of the tokenizer token space. MC is a measure of the local geometric smoothness of the tokenizer token space.
  • Figure 3: Token swapping as a compositionality probe. For each row, we progressively swap an increasing subset of tokens from Image B into Image A. CompTok preserves coherent structure while transferring semantics, e.g., zebra stripes onto a dog; crowd appearance onto an apple, whereas baselines exhibit mixing artifacts and off-manifold blends.
  • Figure 4:
  • Figure 5:
  • ...and 3 more figures

Theorems & Definitions (12)

  • Definition C.1: AvgIG: Usable Information Gain
  • Definition C.2: Local MC: Pairwise Connectivity Ratio
  • Lemma C.3: AvgIG $\equiv$ Likelihood Gradient
  • proof
  • Theorem C.4: Lipschitz Smoothness Lower Bounds MC
  • proof
  • Proposition C.5: MI Maximization Enforces Non-Zero Gradients
  • proof
  • Proposition C.6: Swap Training Bounds Path Loss
  • proof
  • ...and 2 more