Token Maturation: Autoregressive Language Generation via Continuous Token Dynamics

TL;DR

This work rethinks autoregressive language generation by decoupling prediction from discrete token commitment through token maturation: tokens are continuous vectors that evolve in embedding space before discretization, enabling deterministic decoding via argmax and allowing manipulations of the continuous trajectory (e.g., noise, smoothing) without relying on traditional sampling or diffusion. The approach introduces a maturation buffer (the liquid tail), conditioning on noise level and tail length, and a training objective that combines regression with a contrastive loss to prevent collapse and align continuous predictions with discrete identities. A GPT-2–backbone instantiation demonstrates coherent text with sustained uncertainty during maturation, reveals that tail length controls diversity, and shows that classifier-free guidance yields interpretable lookahead in the maturation dynamics. Overall, token maturation offers a new design axis for language modeling that preserves autoregressive causality while enabling stable, geometry-based uncertainty resolution and richer intermediate states during generation.

Abstract

Autoregressive language models are conventionally defined over discrete token sequences, committing to a specific token at every generation step. This early discretization forces uncertainty to be resolved through token-level sampling, often leading to instability, repetition, and sensitivity to decoding heuristics. In this work, we introduce a continuous autoregressive formulation of language generation in which tokens are represented as continuous vectors that \emph{mature} over multiple update steps before being discretized. Rather than sampling tokens, the model evolves continuous token representations through a deterministic dynamical process, committing to a discrete token only when the representation has sufficiently converged. Discrete text is recovered via hard decoding, while uncertainty is maintained and resolved in the continuous space. We show that this maturation process alone is sufficient to produce coherent and diverse text using deterministic decoding (argmax), without reliance on token-level sampling, diffusion-style denoising, or auxiliary stabilization mechanisms. Additional perturbations, such as stochastic dynamics or history smoothing, can be incorporated naturally but are not required for the model to function. To our knowledge, this is the first autoregressive language model that generates text by evolving continuous token representations to convergence prior to discretization, enabling stable generation without token-level sampling.

Figures (5)

• Figure 1: Immediate commitment vs. token maturation. (A) Standard autoregressive decoding commits to a discrete token at each step, making early decisions irreversible. (B) Token maturation maintains a continuous "liquid tail" of token representations that evolve over time; discretization is deferred to a final commitment step.
• Figure 2: Left: Entropy throughout token maturation for four representative tokens. Despite progressing through 24 maturation steps, entropy remains constant at approximately 3.9 nats—the model never collapses to certainty before commitment. Right: Top candidate for token "Dr" at each step, showing exploration through semantically diverse alternatives despite constant entropy. This demonstrates that commitment emerges from geometric convergence, not probability concentration.
• Figure 3: Embedding drift from frozen GPT-2 to learned embeddings. Left: distribution of drift across vocabulary. Right: tokens with highest drift are predominantly punctuation and rare symbols.
• Figure 4: Generation interface showing token maturation in action. Left: Committed text (white) followed by the liquid tail (cyan) containing uncommitted tokens that will mature over subsequent steps. Bottom: Live metrics including entropy ($H=3.91$), confirming sustained uncertainty. Right: Top candidates for the next commitment, showing near-uniform scores over semantically appropriate alternatives (psychology, professor, psychiatrist, neuro). Despite high entropy, all candidates are contextually relevant—the model converges to a semantic region rather than a single token.
• Figure 5: Effect of CFG on tail interpretability. Without CFG (top), tail tokens appear as incoherent noise. With CFG (bottom), tail tokens form semantically meaningful lookahead, revealing the model's implicit forward planning.