Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction

Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock, Kentaro Inui

TL;DR

The paper provides a targeted analysis of GPT-3.5 and GPT-4 on grammatical error correction by exploring fine-grained prompt engineering and evaluating performance on BEA-2019 and JFLEG. It combines automatic metrics (F0.5, GLEU) with human evaluation to reveal strengths in sentence-level revision, particularly for GPT-4 on JFLEG, and highlights clear prompt sensitivity and trade-offs between fluency-centered edits and minimal editing. The study shows that prompt wording, temperature, and few-shot examples significantly shape outputs, with the best prompts producing fluent corrections while sometimes over-editing or expanding content. These findings underscore the importance of task framing and benchmark choice in applying large language models to GEC and related writing-assistance tasks.

Abstract

GPT-3 and GPT-4 models are powerful, achieving high performance on a variety of Natural Language Processing tasks. However, there is a relative lack of detailed published analysis of their performance on the task of grammatical error correction (GEC). To address this, we perform experiments testing the capabilities of a GPT-3.5 model (text-davinci-003) and a GPT-4 model (gpt-4-0314) on major GEC benchmarks. We compare the performance of different prompts in both zero-shot and few-shot settings, analyzing intriguing or problematic outputs encountered with different prompt formats. We report the performance of our best prompt on the BEA-2019 and JFLEG datasets, finding that the GPT models can perform well in a sentence-level revision setting, with GPT-4 achieving a new high score on the JFLEG benchmark. Through human evaluation experiments, we compare the GPT models' corrections to source, human reference, and baseline GEC system sentences and observe differences in editing strategies and how they are scored by human raters.

Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction

TL;DR

The paper provides a targeted analysis of GPT-3.5 and GPT-4 on grammatical error correction by exploring fine-grained prompt engineering and evaluating performance on BEA-2019 and JFLEG. It combines automatic metrics (F0.5, GLEU) with human evaluation to reveal strengths in sentence-level revision, particularly for GPT-4 on JFLEG, and highlights clear prompt sensitivity and trade-offs between fluency-centered edits and minimal editing. The study shows that prompt wording, temperature, and few-shot examples significantly shape outputs, with the best prompts producing fluent corrections while sometimes over-editing or expanding content. These findings underscore the importance of task framing and benchmark choice in applying large language models to GEC and related writing-assistance tasks.

Abstract

GPT-3 and GPT-4 models are powerful, achieving high performance on a variety of Natural Language Processing tasks. However, there is a relative lack of detailed published analysis of their performance on the task of grammatical error correction (GEC). To address this, we perform experiments testing the capabilities of a GPT-3.5 model (text-davinci-003) and a GPT-4 model (gpt-4-0314) on major GEC benchmarks. We compare the performance of different prompts in both zero-shot and few-shot settings, analyzing intriguing or problematic outputs encountered with different prompt formats. We report the performance of our best prompt on the BEA-2019 and JFLEG datasets, finding that the GPT models can perform well in a sentence-level revision setting, with GPT-4 achieving a new high score on the JFLEG benchmark. Through human evaluation experiments, we compare the GPT models' corrections to source, human reference, and baseline GEC system sentences and observe differences in editing strategies and how they are scored by human raters.
Paper Structure (24 sections, 1 equation, 1 figure, 5 tables)

This paper contains 24 sections, 1 equation, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. OpenAI Models
  4. Grammatical Error Correction
  5. Prompt Engineering
  6. Evaluation Experiments
  7. Data and Benchmarks
  8. Human Evaluation
  9. Results
  10. Prompt Engineering
  11. "Meta" information:
  12. Outputs with extra punctuation:
  13. Outputs with spurious expansions:
  14. Benchmark Scoring
  15. Human Evaluation and Subset Analysis
  16. ...and 9 more sections

Figures (1)

  • Figure 1: OpenAI's example prompt for "grammar correction," showing an input and output (highlighted in green) for the sentence-level revision task. Our experiments with GPT-3.5 and GPT-4 are based on this pattern.