Geometry-Aware Decoding with Wasserstein-Regularized Truncation and Mass Penalties for Large Language Models
Arash Gholami Davoodi, Navid Rezazadeh, Seyed Pouyan Mousavi Davoudi, Pouya Pezeshkpour
TL;DR
Top-W introduces geometry-aware decoding for large language models by optimizing a Wasserstein-transport objective over the next-token crop, balancing semantic transport distance, cropped entropy, and retained mass via the fixed-potential framework. The method yields an exact factorization that reduces the S-step to a prefix or singleton search, enabling an efficient O(n) crop update with an anchored Lipschitz potential and a practical top_m candidate pool. An alternating decoder performs iterative f- and S-steps, with a geometry-driven bias anchored to the current crop, and a theoretical and empirical analysis showing connections to Top-$k$ and Top-$H$ under special metrics. Empirically, Top-W achieves robust improvements across reasoning, instruction-following, and creative writing benchmarks with modest runtime overhead, highlighting the value of embedding geometry in decoding and its potential to enhance both accuracy and creativity.
Abstract
Large language models (LLMs) must balance diversity and creativity against logical coherence in open-ended generation. Existing truncation-based samplers are effective but largely heuristic, relying mainly on probability mass and entropy while ignoring semantic geometry of the token space. We present Top-W, a geometry-aware truncation rule that uses Wasserstein distance-defined over token-embedding geometry-to keep the cropped distribution close to the original, while explicitly balancing retained probability mass against the entropy of the kept set. Our theory yields a simple closed-form structure for the fixed-potential subset update: depending on the mass-entropy trade-off, the optimal crop either collapses to a single token or takes the form of a one-dimensional prefix that can be found efficiently with a linear scan. We implement Top-W using efficient geometry-based potentials (nearest-set or k-NN) and pair it with an alternating decoding routine that keeps the standard truncation-and-sampling interface unchanged. Extensive experiments on four benchmarks (GSM8K, GPQA, AlpacaEval, and MT-Bench) across three instruction-tuned models show that Top-W consistently outperforms prior state-of-the-art decoding approaches achieving up to 33.7% improvement. Moreover, we find that Top-W not only improves accuracy-focused performance, but also boosts creativity under judge-based open-ended evaluation.