Think-J: Learning to Think for Generative LLM-as-a-Judge

TL;DR

Think-J tackles the challenge of reliable LLM-based evaluation by teaching judges to think. It initializes judgment thinking traces from a small, curated LIMJ707 dataset and further optimizes thinking via two RL paths: offline critic-guided learning and online rule-based learning, using a thoughtfully designed reward system. Empirical results across RewardBench, RMBench, and Auto-J-test show Think-J, especially with larger bases, surpasses both generative and classifier-based judges with only modest data, while producing interpretable thinking traces and robust performance under data quality variations. This approach offers a scalable, privacy-preserving alternative to external evaluators and demonstrates the value of reasoning traces in preference modeling for LLM evaluation.

Abstract

LLM-as-a-Judge refers to the automatic modeling of preferences for responses generated by Large Language Models (LLMs), which is of significant importance for both LLM evaluation and reward modeling. Although generative LLMs have made substantial progress in various tasks, their performance as LLM-Judge still falls short of expectations. In this work, we propose Think-J, which improves generative LLM-as-a-Judge by learning how to think. We first utilized a small amount of curated data to develop the model with initial judgment thinking capabilities. Subsequently, we optimize the judgment thinking traces based on reinforcement learning (RL). We propose two methods for judgment thinking optimization, based on offline and online RL, respectively. The offline method requires training a critic model to construct positive and negative examples for learning. The online method defines rule-based reward as feedback for optimization. Experimental results showed that our approach can significantly enhance the evaluation capability of generative LLM-Judge, surpassing both generative and classifier-based LLM-Judge without requiring extra human annotations.

Figures (14)

• Figure 1: Comparison of different judge models. Our proposed Think-J takes into account both accuracy and interpretability based on thinking optimization.
• Figure 2: The illustration of our proposed framework. We begin by constructing high-quality judgment thinking traces using curated principles and proprietary thinking models. Based on this data, we initialize a judge model and a critic model, both equipped with judgment thinking capability. After that, we optimize the capability of the judge model through two methods: Critic-guided Offline Learning and Rule-based Online Learning, resulting in Think-J.
• Figure 3: Experiment results of different data source for judgment thinking initialization.
• Figure 4: Comparison of different methods on data groups with different quality. Group 1 is with lower quality and Group 4 is with higher quality.
• Figure 5: The variation of statistical metrics during offline learning trained on Helpsteer2-Pref.