Evaluating Small Decoder-Only Language Models for Grammar Correction and Text Simplification

TL;DR

This study assesses whether small decoder-only language models can match large language models for grammar correction and text simplification. By fine-tuning and cascading several SLMs on BEA-2019/JFLEG and WikiAuto/ASSET, the authors benchmark against GPT-4 and GPT-3.5 Turbo using GLEU, M$^2$, SARI, FRE, and NER-based hallucination metrics. The findings show that while some SLMs achieve competitive results relative to their size, they remain substantially below LLM baselines on grammar correction and struggle with meaning preservation and hallucinations, with cascading often yielding inconsistent gains. The work suggests that, despite efficiency advantages, current SLMs cannot yet replace modern LLMs for high-quality rewriting, and points toward exploring mid-sized models and training innovations as a practical path forward.

Abstract

Large language models have become extremely popular recently due to their ability to achieve strong performance on a variety of tasks, such as text generation and rewriting, but their size and computation cost make them difficult to access, deploy, and secure in many settings. This paper investigates whether small, decoder-only language models can provide an efficient alternative for the tasks of grammar correction and text simplification. The experiments in this paper focus on testing small language models out of the box, fine-tuned, and run sequentially on the JFLEG and ASSET datasets using established metrics. The results show that while SLMs may learn certain behaviors well, their performance remains below strong baselines and current LLMs. The results also show that SLMs struggle with retaining meaning and hallucinations. These findings suggest that despite their efficiency advantages, current SLMs are not yet competitive enough with modern LLMs for rewriting, and further advances in training are required for SLMs to close the performance gap between them and today's LLMs.

Figures (12)

• Figure 1: Single-Model GLEU Scores
• Figure 2: Precision, Recall, and $F_{0.5}$
• Figure 3: Cascade GLEU Scores
• Figure 4: Cascade Precision, Recall, and $F_{0.5}$
• Figure 5: Single-Model SARI Scoring