Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Representation Degeneration Problem in Training Natural Language Generation Models

Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, Tie-Yan Liu

TL;DR

This work identifies a representation degeneration problem in neural natural language generation where word embeddings collapse into a narrow cone under weight tying and likelihood training. It provides a theoretical link between degeneration and hidden-state geometry, particularly for low-frequency words, and proposes MLE-CosReg, a cosine-based regularization to widen embedding dispersion. Empirical results on WikiText-2 language modeling and WMT En-De/De-En machine translation demonstrate consistent perplexity and BLEU improvements with minimal parameter overhead, signaling enhanced embedding expressiveness. The approach offers a practical, complementary tool for improving NLG performance by explicitly shaping embedding geometry.

Abstract

We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the \emph{representation degeneration problem}. We observe that when training a model for natural language generation tasks through likelihood maximization with the weight tying trick, especially with big training datasets, most of the learnt word embeddings tend to degenerate and be distributed into a narrow cone, which largely limits the representation power of word embeddings. We analyze the conditions and causes of this problem and propose a novel regularization method to address it. Experiments on language modeling and machine translation show that our method can largely mitigate the representation degeneration problem and achieve better performance than baseline algorithms.

Representation Degeneration Problem in Training Natural Language Generation Models

TL;DR

This work identifies a representation degeneration problem in neural natural language generation where word embeddings collapse into a narrow cone under weight tying and likelihood training. It provides a theoretical link between degeneration and hidden-state geometry, particularly for low-frequency words, and proposes MLE-CosReg, a cosine-based regularization to widen embedding dispersion. Empirical results on WikiText-2 language modeling and WMT En-De/De-En machine translation demonstrate consistent perplexity and BLEU improvements with minimal parameter overhead, signaling enhanced embedding expressiveness. The approach offers a practical, complementary tool for improving NLG performance by explicitly shaping embedding geometry.

Abstract

We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the \emph{representation degeneration problem}. We observe that when training a model for natural language generation tasks through likelihood maximization with the weight tying trick, especially with big training datasets, most of the learnt word embeddings tend to degenerate and be distributed into a narrow cone, which largely limits the representation power of word embeddings. We analyze the conditions and causes of this problem and propose a novel regularization method to address it. Experiments on language modeling and machine translation show that our method can largely mitigate the representation degeneration problem and achieve better performance than baseline algorithms.

Paper Structure

This paper contains 21 sections, 8 theorems, 13 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Representation Degeneration Problem
  4. Experimental Design
  5. Discussion
  6. Understanding the Problem
  7. Extreme Case: Non-appeared word tokens
  8. Extension to rarely appeared word tokens
  9. Addressing the Problem
  10. Experiments
  11. Experimental Settings
  12. Language Modeling
  13. Machine Translation
  14. Experimental Results
  15. Discussion
  16. ...and 6 more sections

Key Result

Theorem 1

A. If the set of uniformly negative direction is not empty, it is convex. B. If there exists a $v$ that is a uniformly negative direction of $h_i$, $i=1,\cdots,M$, then the optimal solution of Eqn. eq:simplifi-loss satisfies $\parallel w_N^* \parallel=\infty$ and can be achieved by setting $w_N^*=\l

Figures (3)

  • Figure 1: 2D visualization. (a). Visualization of word embeddings trained from vanilla Transformer vaswani2017attention in English$\to$German translation task. (b). Visualization of word embeddings trained from Word2Vec mikolov2013distributed. (c). Visualization of hidden states and category embedding of a classification task, where different colors stand for different categories and the blue triangles denote for category embeddings.
  • Figure 2: (a): Word embeddings trained from MLE-CosReg. (b): Singular values of embedding matrix. We normalize the singular values of each matrix so that the largest one is 1.
  • Figure 3: (a): WMT 2014 English-German Dataset preprocessed with BPE. (b): word-level WikiText-2. In the two figures, the x-axis is the token ranked with respect to its frequency in descending order. The y-axis is the logarithmic value of the token frequency.

Theorems & Definitions (13)

  • Definition 1
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • ...and 3 more