Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language Generation with Strictly Proper Scoring Rules

Chenze Shao, Fandong Meng, Yijin Liu, Jie Zhou

TL;DR

The paper investigates moving beyond the traditional log-likelihood loss by applying strictly proper scoring rules to language generation. It introduces a token-level distribution of non-local scoring rules and a score smoothing mechanism to enable training with the Brier and spherical scores, among others. Empirically, substituting these losses during fine-tuning yields substantial gains on translation and summarization tasks, with notable improvements on large-scale models like LLaMA-7B/13B, and reveals distinct optimization dynamics across scoring rules. The findings suggest potential for multi-score training and new decoding objectives, advancing calibration and generation quality in practical NLP systems.

Abstract

Language generation based on maximum likelihood estimation (MLE) has become the fundamental approach for text generation. Maximum likelihood estimation is typically performed by minimizing the log-likelihood loss, also known as the logarithmic score in statistical decision theory. The logarithmic score is strictly proper in the sense that it encourages honest forecasts, where the expected score is maximized only when the model reports true probabilities. Although many strictly proper scoring rules exist, the logarithmic score is the only local scoring rule among them that depends exclusively on the probability of the observed sample, making it capable of handling the exponentially large sample space of natural text. In this work, we propose a straightforward strategy for adapting scoring rules to language generation, allowing for language modeling with any non-local scoring rules. Leveraging this strategy, we train language generation models using two classic strictly proper scoring rules, the Brier score and the Spherical score, as alternatives to the logarithmic score. Experimental results indicate that simply substituting the loss function, without adjusting other hyperparameters, can yield substantial improvements in model's generation capabilities. Moreover, these improvements can scale up to large language models (LLMs) such as LLaMA-7B and LLaMA-13B. Source code: \url{https://github.com/shaochenze/ScoringRulesLM}.

Language Generation with Strictly Proper Scoring Rules

TL;DR

The paper investigates moving beyond the traditional log-likelihood loss by applying strictly proper scoring rules to language generation. It introduces a token-level distribution of non-local scoring rules and a score smoothing mechanism to enable training with the Brier and spherical scores, among others. Empirically, substituting these losses during fine-tuning yields substantial gains on translation and summarization tasks, with notable improvements on large-scale models like LLaMA-7B/13B, and reveals distinct optimization dynamics across scoring rules. The findings suggest potential for multi-score training and new decoding objectives, advancing calibration and generation quality in practical NLP systems.

Abstract

Language generation based on maximum likelihood estimation (MLE) has become the fundamental approach for text generation. Maximum likelihood estimation is typically performed by minimizing the log-likelihood loss, also known as the logarithmic score in statistical decision theory. The logarithmic score is strictly proper in the sense that it encourages honest forecasts, where the expected score is maximized only when the model reports true probabilities. Although many strictly proper scoring rules exist, the logarithmic score is the only local scoring rule among them that depends exclusively on the probability of the observed sample, making it capable of handling the exponentially large sample space of natural text. In this work, we propose a straightforward strategy for adapting scoring rules to language generation, allowing for language modeling with any non-local scoring rules. Leveraging this strategy, we train language generation models using two classic strictly proper scoring rules, the Brier score and the Spherical score, as alternatives to the logarithmic score. Experimental results indicate that simply substituting the loss function, without adjusting other hyperparameters, can yield substantial improvements in model's generation capabilities. Moreover, these improvements can scale up to large language models (LLMs) such as LLaMA-7B and LLaMA-13B. Source code: \url{https://github.com/shaochenze/ScoringRulesLM}.
Paper Structure (21 sections, 33 equations, 3 figures, 11 tables)

This paper contains 21 sections, 33 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Strictly Proper Scoring Rules
  3. Scoring Rules
  4. Propriety
  5. Locality
  6. Examples
  7. Language Generation with Strictly Proper Scoring Rules
  8. Scoring Rules as Losses
  9. Token-Level Scoring Rules
  10. Score Smoothing
  11. Experiments
  12. Setup
  13. Training from Scratch
  14. Fine-tuning with Scoring Rules
  15. Model Dynamics during Fine-tuning
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Performance curves of different strictly proper scoring rules on translation and summarization tasks.
  • Figure 2: Question answering capabilities evaluated on MT-bench, a multi-turn question set.
  • Figure 3: Performance curves on WMT14 En-De test set when fine-tuning with different scoring rules.