Exploration of Masked and Causal Language Modelling for Text Generation

TL;DR

This work questions the dominant left-to-right CLM paradigm by systematically comparing MLM-based generation against CLM across medical, movie, and authorship domains using synthetic data. It pre-trains and evaluates MLM and CLM models of comparable size, applying diverse masking strategies and assessing generated text with quantitative metrics, human judgments, and three downstream NLP tasks (NER, text classification, authorship verification). Results show MLM consistently surpasses CLM in generation quality, with domain-specific pre-training not yielding reliable gains and no strong link between generation quality and downstream performance. The findings highlight MLM’s potential for high-quality text generation and privacy-preserving synthetic data, while outlining limitations and promising directions for scaling, tasks, and domain expansion in future work.

Abstract

Large Language Models (LLMs) have revolutionised the field of Natural Language Processing (NLP) and have achieved state-of-the-art performance in practically every task in this field. However, the prevalent approach used in text generation, Causal Language Modelling (CLM), which generates text sequentially from left to right, inherently limits the freedom of the model, which does not decide when and where each token is generated. In contrast, Masked Language Modelling (MLM), primarily used for language understanding tasks, can generate tokens anywhere in the text and any order. This paper conducts an extensive comparison of MLM and CLM approaches for text generation tasks. To do so, we pre-train several language models of comparable sizes on three different datasets, namely 1) medical discharge summaries, 2) movie plot synopses, and 3) authorship verification datasets. To assess the quality of the generations, we first employ quantitative metrics and then perform a qualitative human evaluation to analyse coherence and grammatical correctness. In addition, we evaluate the usefulness of the generated texts by using them in three different downstream tasks: 1) Entity Recognition, 2) Text Classification, and 3) Authorship Verification. The results show that MLM consistently outperforms CLM in text generation across all datasets, with higher quantitative scores and better coherence in the generated text. The study also finds \textit{no strong correlation} between the quality of the generated text and the performance of the models in the downstream tasks. With this study, we show that MLM for text generation has great potential for future research and provides direction for future studies in this area.

Figures (15)

• Figure 1: Examples of Inputs and Outputs for MLM models.
• Figure 2: Examples of Inputs and Outputs for CLM models.
• Figure 3: Experimental Design of the Downstream NER Task.
• Figure 4: Experimental Design of the Downstream Text Classification Task.
• Figure 5: Experimental Design of the Downstream Author Verification Task.