Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling

TL;DR

The paper tackles the problem that LLMs struggle to reliably verify long and domain-specific reasoning, which hampers label-efficient training of reasoning models. It introduces data-driven rubrics—granular error taxonomies extracted from unsuccessful reasoning traces—to guide LLM verifiers and to generate improved RL rewards. The authors demonstrate that rubric-informed classifiers achieve up to $11.6\%$ higher trace-correctness identification and enable RL training that approaches the performance of verifiable rewards while using as little as $20\%$ gold labels, across coding, math, and chemical engineering domains. This approach extends the utility of reward rubrics from qualitative assessments to quantitative correctness, enabling domain-adaptive reasoning and reducing reliance on costly gold data. The work holds practical significance for training performant reasoning models in challenging domains with limited labeled data and suggests avenues for further integration with human-AI collaboration.

Abstract

An impediment to using Large Language Models (LLMs) for reasoning output verification is that LLMs struggle to reliably identify errors in thinking traces, particularly in long outputs, domains requiring expert knowledge, and problems without verifiable rewards. We propose a data-driven approach to automatically construct highly granular reasoning error taxonomies to enhance LLM-driven error detection on unseen reasoning traces. Our findings indicate that classification approaches that leverage these error taxonomies, or "rubrics", demonstrate strong error identification compared to baseline methods in technical domains like coding, math, and chemical engineering. These rubrics can be used to build stronger LLM-as-judge reward functions for reasoning model training via reinforcement learning. Experimental results show that these rewards have the potential to improve models' task accuracy on difficult domains over models trained by general LLMs-as-judges by +45%, and approach performance of models trained by verifiable rewards while using as little as 20% as many gold labels. Through our approach, we extend the usage of reward rubrics from assessing qualitative model behavior to assessing quantitative model correctness on tasks typically learned via RLVR rewards. This extension opens the door for teaching models to solve complex technical problems without a full dataset of gold labels, which are often highly costly to procure.

Figures (11)

• Figure 1: We propose using a knowledge store of errors from inference rollouts to enhance LLM-driven reward functions for model training. We pass incorrect reasoning traces alongside ground-truth answers (Box A) to our "rubric generator" that extracts a set of organized failure modes learned from those incorrect traces (Box B). We then pass this rubric to a LLM classifier that identifies whether a reasoning trace will result in an incorrect answer to serve as a reward function (Box C).
• Figure 2: Top: An excerpt from a rubric constructed from chemical engineering problems in the NaturalReasoning dataset, illustrating their hierarchical organization. Bottom: An sample classification made by Claude 3.5 Sonnet using the rubric (showing trace compression, keyword labeling, and rubric item application). The depicted keyword set and rubric items are subsets of the full sets.
• Figure 3: Distribution of most applied rubric item keywords over a technical domain and non-technical (and qualitative) domain, alongside the top most-applied rubric item for each trace set. Both domain problems taken from the Meta NaturalReasoning dataset.
• Figure 4: Experiment 1 ablation: # of rubric items vs. classification specificity.
• Figure 5: Rubric-augmented Claude 3.5 Sonnet outperforms standard prompted Sonnet at identifying incorrect reasoning traces across three diverse domains: SWE-Bench (SB), NuminaMath (NM), and chemical engineering problems from NaturalReasoning (NR). We present balanced accuracy (mean accuracy with correct and incorrect traces weighted equally), specificity (# of incorrect traces classified as incorrect), and F0.5 (measuring precision and recall with additional precision weight).