Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unleashing Large Language Models' Proficiency in Zero-shot Essay Scoring

Sanwoo Lee, Yida Cai, Desong Meng, Ziyang Wang, Yunfang Wu

TL;DR

It is shown that the zero-shot prompting framework, Multi Trait Specialization (MTS), elicits LLMs' ample potential for essay scoring, and the small-sized Llama2-13b-chat substantially outperforms ChatGPT, facilitating an effective deployment in real applications.

Abstract

Advances in automated essay scoring (AES) have traditionally relied on labeled essays, requiring tremendous cost and expertise for their acquisition. Recently, large language models (LLMs) have achieved great success in various tasks, but their potential is less explored in AES. In this paper, we show that our zero-shot prompting framework, Multi Trait Specialization (MTS), elicits LLMs' ample potential for essay scoring. In particular, we automatically decompose writing proficiency into distinct traits and generate scoring criteria for each trait. Then, an LLM is prompted to extract trait scores from several conversational rounds, each round scoring one of the traits based on the scoring criteria. Finally, we derive the overall score via trait averaging and min-max scaling. Experimental results on two benchmark datasets demonstrate that MTS consistently outperforms straightforward prompting (Vanilla) in average QWK across all LLMs and datasets, with maximum gains of 0.437 on TOEFL11 and 0.355 on ASAP. Additionally, with the help of MTS, the small-sized Llama2-13b-chat substantially outperforms ChatGPT, facilitating an effective deployment in real applications.

Unleashing Large Language Models' Proficiency in Zero-shot Essay Scoring

TL;DR

It is shown that the zero-shot prompting framework, Multi Trait Specialization (MTS), elicits LLMs' ample potential for essay scoring, and the small-sized Llama2-13b-chat substantially outperforms ChatGPT, facilitating an effective deployment in real applications.

Abstract

Advances in automated essay scoring (AES) have traditionally relied on labeled essays, requiring tremendous cost and expertise for their acquisition. Recently, large language models (LLMs) have achieved great success in various tasks, but their potential is less explored in AES. In this paper, we show that our zero-shot prompting framework, Multi Trait Specialization (MTS), elicits LLMs' ample potential for essay scoring. In particular, we automatically decompose writing proficiency into distinct traits and generate scoring criteria for each trait. Then, an LLM is prompted to extract trait scores from several conversational rounds, each round scoring one of the traits based on the scoring criteria. Finally, we derive the overall score via trait averaging and min-max scaling. Experimental results on two benchmark datasets demonstrate that MTS consistently outperforms straightforward prompting (Vanilla) in average QWK across all LLMs and datasets, with maximum gains of 0.437 on TOEFL11 and 0.355 on ASAP. Additionally, with the help of MTS, the small-sized Llama2-13b-chat substantially outperforms ChatGPT, facilitating an effective deployment in real applications.
Paper Structure (34 sections, 2 equations, 8 figures, 5 tables)

This paper contains 34 sections, 2 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Multi Trait Decomposition
  4. Trait Specialization
  5. Trait Aggregation and Scaling
  6. Experimental Settings
  7. LLMs
  8. Datasets and Evaluation Metric
  9. Implementation Details
  10. Comparisons with Other Methods
  11. Main Results
  12. Analysis
  13. Analysis of Multi Trait Decomposition
  14. Role of Rubric Guideline and Scoring Criteria.
  15. Leveraging ChatGPT for Guidance Generation.
  16. ...and 19 more sections

Figures (8)

  • Figure 1: Comparison of our MTS zero-shot prompting framework and Vanilla baseline across different types of LLMs and datasets, measured on average QWK.
  • Figure 2: Illustration of the prompt for multi trait decomposition used for ASAP. The contents to be filled are denoted in red. See Appendix \ref{['appendix:mtd']} for the templates used for ASAP and TOEFL11.
  • Figure 3: The illustration of Multi Trait Specialization framework. The parts to be filled with specific contents are substituted with comments between double curly braces and colored in red.
  • Figure 4: Comparison between MTS (Mistral-7b-instruct) and supervised SOTA (NPCR).
  • Figure 5: Ablation over (1) reference to rubric guidelines and (2) scoring criteria generation. The QWKs of Llama2-13b-chat on ASAP are reported.
  • ...and 3 more figures