Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Holistic Scores: Automatic Trait-Based Quality Scoring of Argumentative Essays

Lucile Favero, Juan Antonio Pérez-Ortiz, Tanja Käser, Nuria Oliver

TL;DR

This work investigates trait-based Automatic Argumentative Essay Scoring (AAES) to provide interpretable, rubric-aligned feedback for argumentative writing. It compares two educationally realistic modeling paradigms: (i) structured in-context learning with small open-source LLMs, and (ii) a CORAL-style ordinal BigBird model engineered for long-sequence reasoning. The study demonstrates that explicit modeling of score ordinality yields higher agreement with human raters across five analytic traits, with BigBird-CORAL consistently outperforming all LLM variants and nominal/regression baselines. It also shows that small open-source LLMs, when guided by rubric-aligned prompts, can achieve competitive performance, especially for reasoning-oriented traits, while enabling privacy-preserving deployment. Overall, the results provide actionable insights for designing educational AI systems that deliver transparent, rubric-consistent feedback for argumentative writing, and suggest promising directions in cross-prompt generalization and hybrid assessment frameworks.

Abstract

Automated Essay Scoring systems have traditionally focused on holistic scores, limiting their pedagogical usefulness, especially in the case of complex essay genres such as argumentative writing. In educational contexts, teachers and learners require interpretable, trait-level feedback that aligns with instructional goals and established rubrics. In this paper, we study trait-based Automatic Argumentative Essay Scoring using two complementary modeling paradigms designed for realistic educational deployment: (1) structured in-context learning with small open-source LLMs, and (2) a supervised, encoder-based BigBird model with a CORAL-style ordinal regression formulation, optimized for long-sequence understanding. We conduct a systematic evaluation on the ASAP++ dataset, which includes essay scores across five quality traits, offering strong coverage of core argumentation dimensions. LLMs are prompted with designed, rubric-aligned in-context examples, along with feedback and confidence requests, while we explicitly model ordinality in scores with the BigBird model via the rank-consistent CORAL framework. Our results show that explicitly modeling score ordinality substantially improves agreement with human raters across all traits, outperforming LLMs and nominal classification and regression-based baselines. This finding reinforces the importance of aligning model objectives with rubric semantics for educational assessment. At the same time, small open-source LLMs achieve a competitive performance without task-specific fine-tuning, particularly for reasoning-oriented traits, while enabling transparent, privacy-preserving, and locally deployable assessment scenarios. Our findings provide methodological, modeling, and practical insights for the design of AI-based educational systems that aim to deliver interpretable, rubric-aligned feedback for argumentative writing.

Beyond Holistic Scores: Automatic Trait-Based Quality Scoring of Argumentative Essays

TL;DR

This work investigates trait-based Automatic Argumentative Essay Scoring (AAES) to provide interpretable, rubric-aligned feedback for argumentative writing. It compares two educationally realistic modeling paradigms: (i) structured in-context learning with small open-source LLMs, and (ii) a CORAL-style ordinal BigBird model engineered for long-sequence reasoning. The study demonstrates that explicit modeling of score ordinality yields higher agreement with human raters across five analytic traits, with BigBird-CORAL consistently outperforming all LLM variants and nominal/regression baselines. It also shows that small open-source LLMs, when guided by rubric-aligned prompts, can achieve competitive performance, especially for reasoning-oriented traits, while enabling privacy-preserving deployment. Overall, the results provide actionable insights for designing educational AI systems that deliver transparent, rubric-consistent feedback for argumentative writing, and suggest promising directions in cross-prompt generalization and hybrid assessment frameworks.

Abstract

Automated Essay Scoring systems have traditionally focused on holistic scores, limiting their pedagogical usefulness, especially in the case of complex essay genres such as argumentative writing. In educational contexts, teachers and learners require interpretable, trait-level feedback that aligns with instructional goals and established rubrics. In this paper, we study trait-based Automatic Argumentative Essay Scoring using two complementary modeling paradigms designed for realistic educational deployment: (1) structured in-context learning with small open-source LLMs, and (2) a supervised, encoder-based BigBird model with a CORAL-style ordinal regression formulation, optimized for long-sequence understanding. We conduct a systematic evaluation on the ASAP++ dataset, which includes essay scores across five quality traits, offering strong coverage of core argumentation dimensions. LLMs are prompted with designed, rubric-aligned in-context examples, along with feedback and confidence requests, while we explicitly model ordinality in scores with the BigBird model via the rank-consistent CORAL framework. Our results show that explicitly modeling score ordinality substantially improves agreement with human raters across all traits, outperforming LLMs and nominal classification and regression-based baselines. This finding reinforces the importance of aligning model objectives with rubric semantics for educational assessment. At the same time, small open-source LLMs achieve a competitive performance without task-specific fine-tuning, particularly for reasoning-oriented traits, while enabling transparent, privacy-preserving, and locally deployable assessment scenarios. Our findings provide methodological, modeling, and practical insights for the design of AI-based educational systems that aim to deliver interpretable, rubric-aligned feedback for argumentative writing.
Paper Structure (21 sections, 1 figure, 2 tables)

This paper contains 21 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. Automatic Argumentative Essay Scoring (AAES).
  5. Supervised neural approaches for AES.
  6. LLM-based approaches for AAES.
  7. Method
  8. Dataset
  9. LLMs for AAES
  10. Model configuration and inference settings.
  11. In-Context Learning (ICL).
  12. BigBird CORAL for AAES
  13. CORAL: Ordinal formulation.
  14. Training objective and class imbalance.
  15. Data splits.
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Overview of the two trait-based AAES approaches studied in this paper.