Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training

Abstract

Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.

Figures (27)

• Figure 1: Illustration of our synthetic experiment setting (left). In the middle, we show that a Llama-3.1-8B policy trained with a fine-tuned Qwen3-4B reasoning judge can achieve strong performance under the gold-standard judge gpt-oss-120b's evaluation, while the policy trained with a fine-tuned Qwen3-14B non-reasoning judge cannot and exhibits severe reward hacking. The table on the right shows results on the creative writing subset of Arena-Hard-V2. The Llama-3.1-8B policy trained with the Qwen3-4B reasoning judge is able to achieve superior performance by learning to generate highly effective adversarial outputs.
• Figure 2: Performance of various LLM-judges based on their agreement (Krippendorff's Alpha) with the gold-standard judge, gpt-oss-120b. The LLM-judges are all based on Qwen3 models and grouped by their sizes. Both the base judges and fine-tuned judges are evaluated, and non-reasoning judges and reasoning judges are compared.
• Figure 3: Performance comparison of policies trained with non-reasoning judges of different sizes under the gold-standard judge's evaluation. The sub-figures show policies trained from different initial LLMs.
• Figure 4: Performance of policies trained with non-reasoning judges of different sizes evaluated by the judges used in training.
• Figure 5: Performance of policies trained with Qwen3-4B and Qwen3-8B reasoning judges. The policy performance at different training steps on the test set is shown when evaluated by the training judges (a) and by the gold-standard judge gpt-oss-120b (b).