MAGIC: Multi-Agent Argumentation and Grammar Integrated Critiquer

TL;DR

MAGIC introduces a zero-shot, multi-agent framework for automated essay scoring and feedback (AES/AEF) that decomposes evaluation into five trait-specific agents (prompt adherence, persuasiveness, organization, vocabulary, grammar) with an orchestrator to synthesize holistic scores. Using a GRE Analytical Writing dataset with ground-truth per-trait scores, MAGIC achieves substantial to near-perfect agreement with human raters and outperforms single-agent baselines across multiple open-weight LLMs. The framework also supports detailed, trait-targeted feedback whose quality approaches human feedback, with scalable evaluation via an LLM judge. These results demonstrate the potential for interpretable, high-quality feedback at college level and highlight MAGIC as a practical, extensible approach for classroom use and beyond.

Abstract

Automated Essay Scoring (AES) and Automatic Essay Feedback (AEF) systems aim to reduce the workload of human raters in educational assessment. However, most existing systems prioritize numerical scoring accuracy over feedback quality and are primarily evaluated on pre-secondary school level writing. This paper presents Multi-Agent Argumentation and Grammar Integrated Critiquer (MAGIC), a framework using five specialized agents to evaluate prompt adherence, persuasiveness, organization, vocabulary, and grammar for both holistic scoring and detailed feedback generation. To support evaluation at the college level, we collated a dataset of Graduate Record Examination (GRE) practice essays with expert-evaluated scores and feedback. MAGIC achieves substantial to near-perfect scoring agreement with humans on the GRE data, outperforming baseline LLM models while providing enhanced interpretability through its multi-agent approach. We also compare MAGIC's feedback generation capabilities against ground truth human feedback and baseline models, finding that MAGIC achieves strong feedback quality and naturalness.

Figures (6)

• Figure 1: MAGIC AES Feedback and Scoring Pipeline. Each agent (prompt adherence, persuasiveness, organization, vocabulary, and grammar) scores the essay separately and provides feedback for their assigned trait. The orchestrator merges the agents' results into a holistic score and combined feedback.
• Figure 2: Comparison of holistic score QWK between taking the average across traits (Trait-wise QWK), adding an additional outlier clipping and scaling stage (+Outlier clipping QWK), as shown in lee_unleashing_2024, and using an orchestrator agent (Orchestrator QWK).
• Figure 3: Per-trait score distributions for Gemma 3 27B LLM and human annotated ground-truth on the GRE dataset. Quartiles highlighted as dotted lines.
• Figure 4: Per-trait QWK of MAGIC independent agents across different base LLMs. Our writing dimension traits (T1--T5) are as described in our methodology.
• Figure 5: Head-to-head Model Feedback Win-rates as Rated by a Judge LLM (o4-mini). Value at row $i$ and column $j$ denotes the average win-rate of row $i$ over column $j$ across all 5 criteria (C1--C5).