Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language Model Decoding as Direct Metrics Optimization

Haozhe Ji, Pei Ke, Hongning Wang, Minlie Huang

TL;DR

This paper introduces Daemon, a decoding framework that treats text generation as an information-projection problem: find a decoding distribution $q$ that stays close to the base LM distribution $p_\theta$ while exactly matching the expected scores of multiple human-aligned metrics. The authors derive an analytic energy-based form $p_{\theta,\boldsymbol{\mu}}(\boldsymbol{x}) \propto p_\theta(\boldsymbol{x}) \exp[-E_{\boldsymbol{\mu}}(\boldsymbol{x})]$ with $E_{\boldsymbol{\mu}}(\boldsymbol{x})=\boldsymbol{\mu}^\top \boldsymbol{f}(\boldsymbol{x})$, and solve for $\boldsymbol{\mu}$ via Weighted Importance Sampling, followed by conditional Sampling-Importance-Resampling (SIR) to draw from the globally normalized distribution. Theoretical results guarantee that the optimal decoding distribution improves perplexity relative to the original LM, and experiments across Wikipedia and Wikinews show Daemon consistently outperforms strong baselines on repetition, coherence, diversity, information content, and MAUVE, with supportive human judgments. The approach provides a principled, scalable route to align open-ended generation with human preferences by enforcing multi-criterion metric constraints during decoding. Potential extensions include broader constraint types, integration with RLHF, and more efficient EBM sampling.

Abstract

Despite the remarkable advances in language modeling, current mainstream decoding methods still struggle to generate texts that align with human texts across different aspects. In particular, sampling-based methods produce less-repetitive texts which are often disjunctive in discourse, while search-based methods maintain topic coherence at the cost of increased repetition. Overall, these methods fall short in achieving holistic alignment across a broad range of aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts measured by multiple metrics of desired aspects simultaneously. The resulting decoding distribution enjoys an analytical solution that scales the input language model distribution via a sequence-level energy function defined by these metrics. And most importantly, we prove that this induced distribution is guaranteed to improve the perplexity on human texts, which suggests a better approximation to the underlying distribution of human texts. To facilitate tractable sampling from this globally normalized distribution, we adopt the Sampling-Importance-Resampling technique. Experiments on various domains and model scales demonstrate the superiority of our method in metrics alignment with human texts and human evaluation over strong baselines.

Language Model Decoding as Direct Metrics Optimization

TL;DR

This paper introduces Daemon, a decoding framework that treats text generation as an information-projection problem: find a decoding distribution that stays close to the base LM distribution while exactly matching the expected scores of multiple human-aligned metrics. The authors derive an analytic energy-based form with , and solve for via Weighted Importance Sampling, followed by conditional Sampling-Importance-Resampling (SIR) to draw from the globally normalized distribution. Theoretical results guarantee that the optimal decoding distribution improves perplexity relative to the original LM, and experiments across Wikipedia and Wikinews show Daemon consistently outperforms strong baselines on repetition, coherence, diversity, information content, and MAUVE, with supportive human judgments. The approach provides a principled, scalable route to align open-ended generation with human preferences by enforcing multi-criterion metric constraints during decoding. Potential extensions include broader constraint types, integration with RLHF, and more efficient EBM sampling.

Abstract

Despite the remarkable advances in language modeling, current mainstream decoding methods still struggle to generate texts that align with human texts across different aspects. In particular, sampling-based methods produce less-repetitive texts which are often disjunctive in discourse, while search-based methods maintain topic coherence at the cost of increased repetition. Overall, these methods fall short in achieving holistic alignment across a broad range of aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts measured by multiple metrics of desired aspects simultaneously. The resulting decoding distribution enjoys an analytical solution that scales the input language model distribution via a sequence-level energy function defined by these metrics. And most importantly, we prove that this induced distribution is guaranteed to improve the perplexity on human texts, which suggests a better approximation to the underlying distribution of human texts. To facilitate tractable sampling from this globally normalized distribution, we adopt the Sampling-Importance-Resampling technique. Experiments on various domains and model scales demonstrate the superiority of our method in metrics alignment with human texts and human evaluation over strong baselines.
Paper Structure (42 sections, 4 theorems, 20 equations, 7 figures, 15 tables, 2 algorithms)

This paper contains 42 sections, 4 theorems, 20 equations, 7 figures, 15 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Method: Decoding as Direct Metrics Optimization
  3. Formulation of the Optimization Problem
  4. Theoretical Improvement in Perplexity
  5. Decoding from the Optimal Solution
  6. Coefficients Estimation
  7. Conditional Sampling from EBM
  8. Experiment
  9. Datasets
  10. Evaluation Metric Settings
  11. Baselines and Implementation Details
  12. Human Evaluation
  13. Main Results
  14. Ablation Study
  15. Conclusion and Future Work
  16. ...and 27 more sections

Key Result

Proposition 1

The distribution that solves the optimization problem (optim_problem:min) is in the form of: where $E_{\boldsymbol{\mu}}(\boldsymbol{x})=\boldsymbol{\mu}^\top \boldsymbol{f}(\boldsymbol{x})$ and $S(p)=\{\boldsymbol{x}: p(\boldsymbol{x})>0\}$ is the support of distribution $p$. $\boldsymbol{\mu}\in \mathbb{R}^K$ is determined by the constraints in (optim_problem:min).

Figures (7)

  • Figure 1: The decoding distribution $p_{\theta,\boldsymbol{\mu}}$ induced by Daemon scales the input LM distribution $p_\theta$ with a sequence-level energy function $E_{\boldsymbol{\mu}}$, which leads to a more accurate recovery of the underlying data distribution $p_d$.
  • Figure 2: coh versus div when tuning the temperature $\tau$ of the proposal model of Daemon and hyper-parameters of other baselines.
  • Figure 3: Ablation results of varying the number of candidates for resampling ($M$). Results on the five metrics are compared with the reference and the latency is relative to Greedy decoding.
  • Figure 4: The mean-seeking behavior of forward KL and the mode-seeking behavior of reverse KL.
  • Figure 5: Optimization curve of each $\mu_i$ with three different initializations: zero, randn, rand.
  • ...and 2 more figures

Theorems & Definitions (7)

  • Proposition 1
  • Proposition 2
  • proof : Proof sketch
  • Proposition 2
  • proof
  • Proposition 2
  • proof