Auto-Rubric: Learning to Extract Generalizable Criteria for Reward Modeling

TL;DR

Auto-Rubric introduces a training-free framework that learns generalizable evaluation rubrics from a small set of human preferences and organizes them into a hierarchical Theme-Tips structure. It reframes reward modeling as rubric learning and uses a two-stage process: per-query rubric generation via a Propose-Evaluate-Revise loop, followed by query-agnostic aggregation by maximizing a coding-rate objective. The method achieves state-of-the-art results across multiple reward-modeling benchmarks with remarkable data efficiency (70 preferences, 1.5% of the source data) and demonstrates strong cross-model generalization, including transfer to GPT-4o. By analyzing rubric utility with Coverage, Precision, and Contribution metrics, the work provides interpretable, non-redundant evaluation criteria that improve transparency, robustness, and scalability in LLM alignment.

Abstract

Reward models are essential for aligning Large Language Models (LLMs) with human values, yet their development is hampered by costly preference datasets and poor interpretability. While recent rubric-based approaches offer transparency, they often lack systematic quality control and optimization, creating a trade-off between scalability and reliability. We address these limitations with a novel, training-free framework built on a key assumption: \textit{evaluation rubrics underlying human preferences exhibit significant generalization ability across diverse queries}, a property that enables remarkable data efficiency. Our two-stage approach first infers high-quality, query-specific rubrics using a validation-guided \textbf{Propose-Evaluate-Revise} pipeline. Second, it generalizes these granular rubrics into a compact, non-redundant core set by maximizing an \textbf{information-theoretic coding rate}. The final output is an interpretable, hierarchical "Theme-Tips" rubric set. Extensive experiments demonstrate the framework's exceptional data efficiency and performance. Critically, using just 70 preference pairs (1.5\% of the source data), our method also empowers smaller models like Qwen3-8B to outperform specialized, fully-trained counterparts. This work pioneers a scalable, interpretable, and data-efficient path for reward modeling.

Figures (13)

• Figure 1: Comparison between traditional reward model training and our automatic rubric extraction method. Traditional methods require large-scale preference data and costly training to produce reward models. In contrast, our method uses a small set of representative data to automatically extract interpretable "Theme-Tips" rubrics for reward modeling, achieving low cost and high interpretability.
• Figure 2: Overview of our two-stage rubric extraction framework. Stage 1 (Query-Specific Rubric Inference): An iterative Propose-Evaluate-Revise pipeline generates high-quality, query-specific rubrics from batches of preference data. Stage 2 (Query-Agnostic Rubric Aggregation): An information-theoretic selection algorithm distills the accumulated rubric pool into a compact, non-redundant core set by maximizing coding rate. The final output is structured into an interpretable "Theme-Tips" hierarchy for enhanced usability and generalization.
• Figure 3: t-SNE Visualization of Selection Order
• Figure 4: Information Gain per Batch
• Figure 6: RewardBench2 Performance