Distribution-Conditional Generation: From Class Distribution to Creative Generation

TL;DR

This work reframes creativity in text-to-image diffusion as distribution-conditioned generation, where images are synthesized from class distributions rather than fixed prompts. It introduces DisTok, an encoder-decoder system that maps distributions to creative concept tokens and builds them via a growing Concept Pool, combining continuous concept fusion with distribution-consistency supervision from a vision-language model. The approach yields state-of-the-art results in distribution-conditional generation, text-pair fusion, and unconditional creative generation while delivering significant speedups and strong semantic alignment, as validated by quantitative metrics, GPT-4o evaluations, and human studies. By enabling open-ended, style-robust concept generation without extensive optimization, DisTok offers a scalable framework for controllable and imaginative visual synthesis.

Abstract

Text-to-image (T2I) diffusion models are effective at producing semantically aligned images, but their reliance on training data distributions limits their ability to synthesize truly novel, out-of-distribution concepts. Existing methods typically enhance creativity by combining pairs of known concepts, yielding compositions that, while out-of-distribution, remain linguistically describable and bounded within the existing semantic space. Inspired by the soft probabilistic outputs of classifiers on ambiguous inputs, we propose Distribution-Conditional Generation, a novel formulation that models creativity as image synthesis conditioned on class distributions, enabling semantically unconstrained creative generation. Building on this, we propose DisTok, an encoder-decoder framework that maps class distributions into a latent space and decodes them into tokens of creative concept. DisTok maintains a dynamic concept pool and iteratively sampling and fusing concept pairs, enabling the generation of tokens aligned with increasingly complex class distributions. To enforce distributional consistency, latent vectors sampled from a Gaussian prior are decoded into tokens and rendered into images, whose class distributions-predicted by a vision-language model-supervise the alignment between input distributions and the visual semantics of generated tokens. The resulting tokens are added to the concept pool for subsequent composition. Extensive experiments demonstrate that DisTok, by unifying distribution-conditioned fusion and sampling-based synthesis, enables efficient and flexible token-level generation, achieving state-of-the-art performance with superior text-image alignment and human preference scores.

Figures (13)

• Figure 1: (a) Traditional class-conditional generation maps a single label to a concept. (b) Text pair-to-object generation fuses two known concepts into a novel one. (c) In contrast, our proposed distribution-conditional generation accepts arbitrary semantic distributions over multiple classes, enabling controllable, fine-grained creativity beyond simple two-token interpolation.
• Figure 2: Overview of DisTok. At each training step, DisTok performs either (a) concept combination by sampling a class pair to train the decoder to generate tokens aligned with increasingly complex class distributions, or (b) distribution consistency supervision by sampling a class distribution to align the encoder and decoder with the visual semantics of generated tokens. (c) Latent vectors are periodically sampled and decoded into tokens, with class distributions predicted by a vision-language model. Resulting tokens and distributions are saved for subsequent combination and supervision.
• Figure 3: Performance of DisTok on Distribution-Conditional Generation task.
• Figure 4: Performance of DisTok on TP2O task.
• Figure 5: Performance of DisTok in Direct Creative Generation without Reference Concepts.