Confusion-Aware Rubric Optimization for LLM-based Automated Grading

TL;DR

Confusion-Aware Rubric Optimization (CARO) is introduced, a novel framework that enhances accuracy and computational efficiency by structurally separating error signals and suggests that replacing mixed-error aggregation with surgical, mode-specific repair yields robust improvements in automated assessment scalability and precision.

Abstract

Accurate and unambiguous guidelines are critical for large language model (LLM) based graders, yet manually crafting these prompts is often sub-optimal as LLMs can misinterpret expert guidelines or lack necessary domain specificity. Consequently, the field has moved toward automated prompt optimization to refine grading guidelines without the burden of manual trial and error. However, existing frameworks typically aggregate independent and unstructured error samples into a single update step, resulting in "rule dilution" where conflicting constraints weaken the model's grading logic. To address these limitations, we introduce Confusion-Aware Rubric Optimization (CARO), a novel framework that enhances accuracy and computational efficiency by structurally separating error signals. CARO leverages the confusion matrix to decompose monolithic error signals into distinct modes, allowing for the diagnosis and repair of specific misclassification patterns individually. By synthesizing targeted "fixing patches" for dominant error modes and employing a diversity-aware selection mechanism, the framework prevents guidance conflict and eliminates the need for resource-heavy nested refinement loops. Empirical evaluations on teacher education and STEM datasets demonstrate that CARO significantly outperforms existing SOTA methods. These results suggest that replacing mixed-error aggregation with surgical, mode-specific repair yields robust improvements in automated assessment scalability and precision.

Figures (7)

• Figure 1: Framework of the CARO algorithm.
• Figure 2: Example Reflector prompt for analyzing the dominant error mode $(0 \to 1)$. The prompt provides: (1) global context via the full confusion matrix, (2) local error examples with model reasoning traces, and (3) contrastive correct examples to establish decision boundaries.
• Figure 3: Structured diagnosis output from the Reflector for mode $(0 \to 1)$. The output identifies root causes, misleading patterns, and proposes targeted rule modifications with safety considerations.
• Figure 4: Refiner prompt structure showing how cross-mode awareness is incorporated. The Refiner receives information about all active error modes to generate compatible rule fixes.
• Figure 5: Two-phase rule consolidation. Phase 1 generates per-mode rules with cross-mode awareness. Phase 2 synthesizes them into a priority-weighted consolidated ruleset with explicit conflict resolution directives.