MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies

TL;DR

MixCE addresses over-generalization in autoregressive language modeling by jointly optimizing forward cross-entropy against the data distribution $P$ and a tractable approximation to the reverse cross-entropy against the model distribution $Q_\theta$. Interpreted as a mixture of forward and reverse KL divergences, MixCE promotes both diversity and text quality, reducing reliance on post-hoc decoding fixes. Across synthetic and real-data GPT-2 finetuning, MixCE yields higher Mauve scores and coherence, and is often preferred in human evaluations, though perplexity can be worse; the gains persist under robustness analyses and controlled metrics. The approach provides a practical, self-reinforced training signal that can be integrated with existing pretrained LMs and software, with code released for reproducibility and potential pretraining use.

Abstract

Autoregressive language models are trained by minimizing the cross-entropy of the model distribution Q relative to the data distribution P -- that is, minimizing the forward cross-entropy, which is equivalent to maximum likelihood estimation (MLE). We have observed that models trained in this way may "over-generalize", in the sense that they produce non-human-like text. Moreover, we believe that reverse cross-entropy, i.e., the cross-entropy of P relative to Q, is a better reflection of how a human would evaluate text generated by a model. Hence, we propose learning with MixCE, an objective that mixes the forward and reverse cross-entropies. We evaluate models trained with this objective on synthetic data settings (where P is known) and real data, and show that the resulting models yield better generated text without complex decoding strategies. Our code and models are publicly available at https://github.com/bloomberg/mixce-acl2023

Figures (12)

• Figure 1: MixCE combines two complementary driving forces: reverse CE helps narrow the model distribution $Q_\theta$ down when it is broader than data distribution $P$, while forward CE helps broaden $Q_\theta$ out when it is narrower than $P$.
• Figure 2: The histograms of sequence-level and token-level negative log-likelihoods of human texts and model generations from GPT-2 large.
• Figure 3: Forward CE only weakly penalizes the model $Q_\theta$ when it puts a small amount of probability mass onto $P(x)=0$ space. And the loss magnitude is much smaller than what we will get from reverse CE.
• Figure 4: The mauve scores obtained by MixCE-finetuned GPT-2 models on development sets with different max generation lengths and different $\eta$. Note that when $\eta=1$, MixCE is equivalent to MLE. The x-axis is the mixing ratio $\eta$, and the y-axis refers to mauve scores with different max generation lengths. The 3 lines in each subplot show the results of GPT-2 models in different sizes. The 3 subplots in each row are the results of 3 datasets respectively. Unbiased sampling is used as the decoding method. Each dot is the average of 3 runs of sampling and the error bar shows the standard deviation of 3 runs.
• Figure 5: Human evaluation interface and a random example from our collected human annotations.