Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder

TL;DR

This work investigates whether decoder-only language models are viable for seq2seq tasks relative to the classic Encoder-Decoder framework. It introduces Regularized Encoder-Decoder (RED) to enable fair, apples-to-apples comparisons and identifies an Attention Degeneration Problem (ADP) where source attention fades as generation proceeds. To address ADP, the authors propose Partial Attention Language Model (PALM), which preserves beneficial LM components while restricting attention to the source through partial attention, separate positional encodings, and language embeddings. Experiments across machine translation, summarization, and data-to-text generation show PALM consistently outperforms LM and approaches ED performance, with ablations validating the contribution of each PALM component.

Abstract

The sequence-to-sequence (seq2seq) task aims at generating the target sequence based on the given input source sequence. Traditionally, most of the seq2seq task is resolved by the Encoder-Decoder framework which requires an encoder to encode the source sequence and a decoder to generate the target text. Recently, a bunch of new approaches have emerged that apply decoder-only language models directly to the seq2seq task. Despite the significant advancements in applying language models to the seq2seq task, there is still a lack of thorough analysis on the effectiveness of the decoder-only language model architecture. This paper aims to address this gap by conducting a detailed comparison between the encoder-decoder architecture and the decoder-only language model framework through the analysis of a regularized encoder-decoder structure. This structure is designed to replicate all behaviors in the classical decoder-only language model but has an encoder and a decoder making it easier to be compared with the classical encoder-decoder structure. Based on the analysis, we unveil the attention degeneration problem in the language model, namely, as the generation step number grows, less and less attention is focused on the source sequence. To give a quantitative understanding of this problem, we conduct a theoretical sensitivity analysis of the attention output with respect to the source input. Grounded on our analysis, we propose a novel partial attention language model to solve the attention degeneration problem. Experimental results on machine translation, summarization, and data-to-text generation tasks support our analysis and demonstrate the effectiveness of our proposed model.

Figures (6)

• Figure 1: Encoder-Decoder (ED) framework and decoder-only Language Model (LM).
• Figure 2: Encoder-Decoder framework (left) and Regularized Encoder-Decoder framework (right).
• Figure 3: PALM framework.
• Figure 4: Numerical sensitivity analysis.
• Figure 5: Stepwise hallucination analysis. Lower is better.

Theorems & Definitions (10)

• Proposition 3.1
• Definition 3.2: Sensitivity
• Theorem 3.3
• Proposition A.1.1
• proof
• Lemma A.2.1
• proof
• Lemma A.2.2
• Theorem A.2.3
• proof