Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness

TL;DR

The work tackles rubric-based short-answer grading with LLM judges, addressing alignment with expert judgments, uncertainty management through consensus deferral, and robustness to linguistic perturbations and adversarial inputs. It introduces a rubric-conditioned grading pipeline that relies on the rubric as primary supervision, augmented by a multi-run consensus mechanism and a small anchor calibration set. Empirical results on the SciEntsBank benchmark with Qwen 2.5-72B reveal strong alignment for binary tasks but notable degradation with 3- and 5-way rubrics; the consensus approach improves accuracy at the cost of reduced coverage and higher compute. The findings emphasize the importance of uncertainty estimation and robustness testing for practical deployment of rubric-conditioned LLM judges in educational settings.

Abstract

Automated short-answer grading (ASAG) remains a challenging task due to the linguistic variability of student responses and the need for nuanced, rubric-aligned partial credit. While Large Language Models (LLMs) offer a promising solution, their reliability as automated judges in rubric-based settings requires rigorous assessment. In this paper, we systematically evaluate the performance of LLM-judges for rubric-based short-answer grading. We investigate three key aspects: the alignment of LLM grading with expert judgment across varying rubric complexities, the trade-off between uncertainty and accuracy facilitated by a consensus-based deferral mechanism, and the model's robustness under random input perturbations and adversarial attacks. Using the SciEntsBank benchmark and Qwen 2.5-72B, we find that alignment is strong for binary tasks but degrades with increased rubric granularity. Our "Trust Curve" analysis demonstrates a clear trade-off where filtering low-confidence predictions improves accuracy on the remaining subset. Additionally, robustness experiments reveal that while the model is resilient to prompt injection, it is sensitive to synonym substitutions. Our work provides critical insights into the capabilities and limitations of rubric-conditioned LLM judges, highlighting the importance of uncertainty estimation and robustness testing for reliable deployment.

Figures (5)

• Figure 1: Confusion matrix comparing model predictions to human gold labels. The diagonal represents correct classifications, while the upper and lower triangles indicate where the model was harsher or more lenient than the human graders, respectively.
• Figure 2: Performance degradation with increasing rubric complexity. Both Accuracy (blue bars) and Cohen's Kappa (red line) decline as the task shifts from binary (2-way) to granular (5-way) grading, illustrating the inverse relationship between label space size and model alignment.
• Figure 3: Selective prediction performance. By raising the consensus threshold (shifting from blue to yellow points), the system filters out uncertain samples.
• Figure 4: Accuracy, Cohen's Kappa score, and Spearman Correlation under Data Augmentation. Augmentation Strategies include character-level noise (ocr, typo, hyphen, non unicode) and semantic/lexical variations (synonym, paraphrase, non influential).
• Figure 5: Distribution of raw scores assigned by the LLM judge (N=500). The left-most column has the unmodified baseline answers, followed by three adversarial categories: Naive inputs (e.g., "solution," "I don't know"), Persuasive injections ("ignore previous answer," quality claims), and Structured attacks (mimicking scoring formats).