Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge

Luyu Chen, Zeyu Zhang, Haoran Tan, Quanyu Dai, Hao Yang, Zhenhua Dong, Xu Chen

TL;DR

This work tackles the limitation of single-point judgments in LLM-based evaluation by explicitly aligning LLM judgment distributions with human evaluation distributions. It introduces a distributional alignment framework that uses a KL-divergence objective combined with an auxiliary cross-entropy loss, and strengthens robustness via adversarial training that perturbs empirical distributions through Projected Gradient Descent. Empirical results across multiple backbones and tasks show reduced distributional misalignment (lower KL) and maintained accuracy, with stronger models achieving closer alignment to human judgments. The approach improves the fidelity, diversity, and robustness of automatic evaluations, offering a scalable alternative to purely human or single-point LLM assessments.

Abstract

LLMs have emerged as powerful evaluators in the LLM-as-a-Judge paradigm, offering significant efficiency and flexibility compared to human judgments. However, previous methods primarily rely on single-point evaluations, overlooking the inherent diversity and uncertainty in human evaluations. This approach leads to information loss and decreases the reliability of evaluations. To address this limitation, we propose a novel training framework that explicitly aligns the LLM-generated judgment distribution with empirical human distributions. Specifically, we propose a distributional alignment objective based on KL divergence, combined with an auxiliary cross-entropy regularization to stabilize the training process. Furthermore, considering that empirical distributions may derive from limited human annotations, we incorporate adversarial training to enhance model robustness against distribution perturbations. Extensive experiments across various LLM backbones and evaluation tasks demonstrate that our framework significantly outperforms existing closed-source LLMs and conventional single-point alignment methods, with improved alignment quality, evaluation accuracy, and robustness.

Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge

TL;DR

This work tackles the limitation of single-point judgments in LLM-based evaluation by explicitly aligning LLM judgment distributions with human evaluation distributions. It introduces a distributional alignment framework that uses a KL-divergence objective combined with an auxiliary cross-entropy loss, and strengthens robustness via adversarial training that perturbs empirical distributions through Projected Gradient Descent. Empirical results across multiple backbones and tasks show reduced distributional misalignment (lower KL) and maintained accuracy, with stronger models achieving closer alignment to human judgments. The approach improves the fidelity, diversity, and robustness of automatic evaluations, offering a scalable alternative to purely human or single-point LLM assessments.

Abstract

LLMs have emerged as powerful evaluators in the LLM-as-a-Judge paradigm, offering significant efficiency and flexibility compared to human judgments. However, previous methods primarily rely on single-point evaluations, overlooking the inherent diversity and uncertainty in human evaluations. This approach leads to information loss and decreases the reliability of evaluations. To address this limitation, we propose a novel training framework that explicitly aligns the LLM-generated judgment distribution with empirical human distributions. Specifically, we propose a distributional alignment objective based on KL divergence, combined with an auxiliary cross-entropy regularization to stabilize the training process. Furthermore, considering that empirical distributions may derive from limited human annotations, we incorporate adversarial training to enhance model robustness against distribution perturbations. Extensive experiments across various LLM backbones and evaluation tasks demonstrate that our framework significantly outperforms existing closed-source LLMs and conventional single-point alignment methods, with improved alignment quality, evaluation accuracy, and robustness.
Paper Structure (20 sections, 11 equations, 3 figures, 3 tables)

This paper contains 20 sections, 11 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-as-a-Judge
  4. Distributional Reward Models
  5. Adversarial Training
  6. Preliminary
  7. Methodology
  8. Overview
  9. Human Distribution Alignment via Hybrid Loss
  10. Robust Alignment via Adversarial Training
  11. Implementing Adversarial Training via Projected Gradient Descent
  12. Experiments
  13. Experiment Setup
  14. Overall Performance
  15. Ablation Study
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Comparison between single-point alignment and distribution alignment. (a) Single-point Alignment: in this method, LLMs are trained to generate outputs that exactly match the desired text. (b) Distribution Alignment: by using this approach, the models are trained to produce judgment distributions that align with the human evaluation distributions.
  • Figure 2: Overview of our proposed framework. (a) Training framework: We generate adversarial perturbations of the empirical human distribution and optimize the hybrid loss. (b) Motivation: Illustrates the relationship between the empirical, perturbed, and true underlying distributions. Robust alignment mitigates the deviation problem in the empirical human distribution.
  • Figure 3: Effect of weighting parameter $\alpha$ and perturbation radius $\epsilon$ on KL divergence across four datasets. Lower values indicate better alignment between model predictions and human distributions.