Pretrained Language Models for Text Generation: A Survey

TL;DR

This survey comprehensively analyzes how pretrained language models are leveraged for text generation. It covers modeling of diverse input types (unstructured, structured, multimedia), strategies to ensure output properties such as relevance, faithfulness, and order preservation, and a range of fine-tuning techniques from data, task, and model perspectives. The authors synthesize major approaches (encoder-decoder vs. decoder-only PLMs, cross-modal pretraining, and grounding) and discuss practical directions like knowledge integration, controllable generation, and compression, while highlighting ethical considerations. Overall, the work provides a structured roadmap for researchers to design PLM-based generation systems and identify gaps for future work.

Abstract

Text generation has become one of the most important yet challenging tasks in natural language processing (NLP). The resurgence of deep learning has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs). In this paper, we present an overview of the major advances achieved in the topic of PLMs for text generation. As the preliminaries, we present the general task definition and briefly describe the mainstream architectures of PLMs for text generation. As the core content, we discuss how to adapt existing PLMs to model different input data and satisfy special properties in the generated text. We further summarize several important fine-tuning strategies for text generation. Finally, we present several future directions and conclude this paper. Our survey aims to provide text generation researchers a synthesis and pointer to related research.