J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge

TL;DR

This work addresses the challenge of interpretable, scalable evaluation for AI by introducing J1-7B, an LLM-as-a-Judge trained with a two-stage process: reflection-enhanced supervised fine-tuning via rejection sampling and reinforcement learning with verifiable rewards. It couples this training with Simple Test-Time Scaling, which inserts reflective tokens (e.g., ``wait'') during inference to extend reasoning and improve judgment accuracy. Empirically, J1-7B surpasses prior LLM-as-a-Judge baselines by 4.8% and exhibits a 5.1% stronger STTS scaling trend, with evidence that the STTS capability emerges primarily during RL rather than SFT alone. The results suggest that structured reflection, guided by RL, enables more reliable and scalable evaluation of complex reasoning tasks, offering a practical path toward improved AI oversight and alignment.

Abstract

The current focus of AI research is shifting from emphasizing model training towards enhancing evaluation quality, a transition that is crucial for driving further advancements in AI systems. Traditional evaluation methods typically rely on reward models assigning scalar preference scores to outputs. Although effective, such approaches lack interpretability, leaving users often uncertain about why a reward model rates a particular response as high or low. The advent of LLM-as-a-Judge provides a more scalable and interpretable method of supervision, offering insights into the decision-making process. Moreover, with the emergence of large reasoning models, which consume more tokens for deeper thinking and answer refinement, scaling test-time computation in the LLM-as-a-Judge paradigm presents an avenue for further boosting performance and providing more interpretability through reasoning traces. In this paper, we introduce $\textbf{J1-7B}$, which is first supervised fine-tuned on reflection-enhanced datasets collected via rejection-sampling and subsequently trained using Reinforcement Learning (RL) with verifiable rewards. At inference time, we apply Simple Test-Time Scaling (STTS) strategies for additional performance improvement. Experimental results demonstrate that $\textbf{J1-7B}$ surpasses the previous state-of-the-art LLM-as-a-Judge by $ \textbf{4.8}$\% and exhibits a $ \textbf{5.1}$\% stronger scaling trend under STTS. Additionally, we present three key findings: (1) Existing LLM-as-a-Judge does not inherently exhibit such scaling trend. (2) Model simply fine-tuned on reflection-enhanced datasets continues to demonstrate similarly weak scaling behavior. (3) Significant scaling trend emerges primarily during the RL phase, suggesting that effective STTS capability is acquired predominantly through RL training.

Figures (27)

• Figure 1: Comparison of Bradley-Terry model and LLM-as-a-Judge under different scaling strategies.
• Figure 2: Pipeline of J1-7B. We first curate the SFT dataset through rejection sampling and subsequently apply RL training to integrate STTS capabilities into J1-7B .
• Figure 3: Scaling trend for STTS on four different tasks.
• Figure 4: Cold start on reasoning-intensive data improves STTS.
• Figure 5: Scaling trend for STTS on Anthropic Harmless and CodePrefBench with different RL algorithms.