Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading

Jamiu Adekunle Idowu, Ahmed Almasoud

TL;DR

This study probes how architectural choices in LLM-based automated essay scoring influence performance across essay quality levels. By comparing a single-agent baseline with a multi-agent system (Content, Structure, Language specialists) coordinated by a rubric-aware Chairman, and by evaluating zero-shot and few-shot prompting on ASAP 2.0, the authors show that few-shot calibration is the dominant driver of performance. The multi-agent approach excels at detecting very low-quality essays due to its dimension-specific veto/capping rules, while the single-agent model tends to perform better on mid-range essays, with both struggling on high-quality essays. The findings offer practical guidance for deploying AES: multi-agent architectures are advantageous for diagnostic screening and interpretability, but come at higher computational cost, whereas single-agent models provide a cost-effective solution for general assessment. Calibration strategy and domain-specific aggregation emerge as key levers for robust, rubric-aligned automate scoring.

Abstract

Automated essay scoring (AES) systems increasingly rely on large language models, yet little is known about how architectural choices shape their performance across different essay quality levels. This paper evaluates single-agent and multi-agent LLM architectures for essay grading using the ASAP 2.0 corpus. Our multi-agent system decomposes grading into three specialist agents (Content, Structure, Language) coordinated by a Chairman Agent that implements rubric-aligned logic including veto rules and score capping. We test both architectures in zero-shot and few-shot conditions using GPT-5.1. Results show that the multi-agent system is significantly better at identifying weak essays while the single-agent system performs better on mid-range essays. Both architectures struggle with high-quality essays. Critically, few-shot calibration emerges as the dominant factor in system performance -- providing just two examples per score level improves QWK by approximately 26% for both architectures. These findings suggest architectural choice should align with specific deployment priorities, with multi-agent AI particularly suited for diagnostic screening of at-risk students, while single-agent models provide a cost-effective solution for general assessment.

Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading

TL;DR

This study probes how architectural choices in LLM-based automated essay scoring influence performance across essay quality levels. By comparing a single-agent baseline with a multi-agent system (Content, Structure, Language specialists) coordinated by a rubric-aware Chairman, and by evaluating zero-shot and few-shot prompting on ASAP 2.0, the authors show that few-shot calibration is the dominant driver of performance. The multi-agent approach excels at detecting very low-quality essays due to its dimension-specific veto/capping rules, while the single-agent model tends to perform better on mid-range essays, with both struggling on high-quality essays. The findings offer practical guidance for deploying AES: multi-agent architectures are advantageous for diagnostic screening and interpretability, but come at higher computational cost, whereas single-agent models provide a cost-effective solution for general assessment. Calibration strategy and domain-specific aggregation emerge as key levers for robust, rubric-aligned automate scoring.

Abstract

Automated essay scoring (AES) systems increasingly rely on large language models, yet little is known about how architectural choices shape their performance across different essay quality levels. This paper evaluates single-agent and multi-agent LLM architectures for essay grading using the ASAP 2.0 corpus. Our multi-agent system decomposes grading into three specialist agents (Content, Structure, Language) coordinated by a Chairman Agent that implements rubric-aligned logic including veto rules and score capping. We test both architectures in zero-shot and few-shot conditions using GPT-5.1. Results show that the multi-agent system is significantly better at identifying weak essays while the single-agent system performs better on mid-range essays. Both architectures struggle with high-quality essays. Critically, few-shot calibration emerges as the dominant factor in system performance -- providing just two examples per score level improves QWK by approximately 26% for both architectures. These findings suggest architectural choice should align with specific deployment priorities, with multi-agent AI particularly suited for diagnostic screening of at-risk students, while single-agent models provide a cost-effective solution for general assessment.
Paper Structure (22 sections, 4 figures, 3 tables)

This paper contains 22 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Capabilities in Automated Scoring
  4. Multi-Agent Architectures in Education
  5. The Architectural Trade-off: Single vs. Multi-Agent
  6. Methodology
  7. Dataset
  8. Experiment
  9. Baseline: Single-Agent Architecture
  10. Multi-Agent Architecture
  11. Evaluation
  12. Results and Discussion
  13. Performance by Score Level
  14. Low-Quality Essays (Scores 1-2)
  15. Mid-Range Essays (Scores 3-4)
  16. ...and 7 more sections

Figures (4)

  • Figure 1: High-level overview of the methodology
  • Figure 2: The structural composition of the multi-agent system.
  • Figure 3: Performance comparison: QWK Score
  • Figure 4: Accuracy across score levels