SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models

TL;DR

This work tackles the problem of evaluating LLM-based subjective answer scoring (SAS) by introducing SAS-Bench, a fine-grained benchmark that supports step-wise scoring and explainability. Built from Gaokao-derived questions across nine subjects, it contains 1,030 questions and 4,109 expert-annotated student responses, with an explicit error-cause taxonomy and a rule bank guiding evaluation. The framework defines two metrics, $CCS$ and $ECS$, to assess holistic and step-wise scoring consistency and error-cause interpretation across 16 LLMs, revealing that few-shot demonstrations and clear scoring guidelines improve performance while step-wise reasoning in science subjects remains challenging. SAS-Bench thus provides a robust benchmark to develop fairer, more transparent, and educationally meaningful LLM-based SAS systems.

Abstract

Subjective Answer Grading (SAG) plays a crucial role in education, standardized testing, and automated assessment systems, particularly for evaluating short-form responses in Short Answer Scoring (SAS). However, existing approaches often produce coarse-grained scores and lack detailed reasoning. Although large language models (LLMs) have demonstrated potential as zero-shot evaluators, they remain susceptible to bias, inconsistencies with human judgment, and limited transparency in scoring decisions. To overcome these limitations, we introduce SAS-Bench, a benchmark specifically designed for LLM-based SAS tasks. SAS-Bench provides fine-grained, step-wise scoring, expert-annotated error categories, and a diverse range of question types derived from real-world subject-specific exams. This benchmark facilitates detailed evaluation of model reasoning processes and explainability. We also release an open-source dataset containing 1,030 questions and 4,109 student responses, each annotated by domain experts. Furthermore, we conduct comprehensive experiments with various LLMs, identifying major challenges in scoring science-related questions and highlighting the effectiveness of few-shot prompting in improving scoring accuracy. Our work offers valuable insights into the development of more robust, fair, and educationally meaningful LLM-based evaluation systems.

Figures (11)

• Figure 1: Comparison of existing benchmarks and our benchmark.
• Figure 1: Statistics of our dataset. "Avg. Steps" and "Max Steps" denote the average and maximum number of steps per response, respectively. "Avg. Length" denotes the average length of per response.
• Figure 2: The workflow of our SAS-Bench. The results from the human judger are predefined during the dataset construction.
• Figure 3: Distribution of question types across different subjects.
• Figure 4: Comparison of QWK scores across LLMs.