Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems

Jiaxin Gao, Chen Chen, Yanwen Jia, Xueluan Gong, Kwok-Yan Lam, Qian Wang

TL;DR

This work analyzes bias in LLM-based judges evaluating AI-generated content for communication systems, focusing on pointwise scoring across 11 bias types. It uses GPT-Judge and JudgeLM with two prompting regimes to assess how biases affect scores over multiple datasets (MMLU-Pro, GPQA, JudgeLM-val) and demonstrates that a detailed rubric reduces bias sensitivity while biased training data can deteriorate performance. The study finds that factual errors cause the largest score degradation, whereas some implicit biases have milder effects, and that prompt design significantly influences outcomes. Four mitigation strategies are proposed—robust prompts, automated bias detection, model calibration with domain-specific training, and ensemble/human-in-the-loop oversight—aimed at enabling fair, trustworthy AI evaluation in telecom and network management contexts.

Abstract

Large Language Models (LLMs) are increasingly being used to autonomously evaluate the quality of content in communication systems, e.g., to assess responses in telecom customer support chatbots. However, the impartiality of these AI "judges" is not guaranteed, and any biases in their evaluation criteria could skew outcomes and undermine user trust. In this paper, we systematically investigate judgment biases in two LLM-as-a-judge models (i.e., GPT-Judge and JudgeLM) under the point-wise scoring setting, encompassing 11 types of biases that cover both implicit and explicit forms. We observed that state-of-the-art LLM judges demonstrate robustness to biased inputs, generally assigning them lower scores than the corresponding clean samples. Providing a detailed scoring rubric further enhances this robustness. We further found that fine-tuning an LLM on high-scoring yet biased responses can significantly degrade its performance, highlighting the risk of training on biased data. We also discovered that the judged scores correlate with task difficulty: a challenging dataset like GPQA yields lower average scores, whereas an open-ended reasoning dataset (e.g., JudgeLM-val) sees higher average scores. Finally, we proposed four potential mitigation strategies to ensure fair and reliable AI judging in practical communication scenarios.

Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems

TL;DR

This work analyzes bias in LLM-based judges evaluating AI-generated content for communication systems, focusing on pointwise scoring across 11 bias types. It uses GPT-Judge and JudgeLM with two prompting regimes to assess how biases affect scores over multiple datasets (MMLU-Pro, GPQA, JudgeLM-val) and demonstrates that a detailed rubric reduces bias sensitivity while biased training data can deteriorate performance. The study finds that factual errors cause the largest score degradation, whereas some implicit biases have milder effects, and that prompt design significantly influences outcomes. Four mitigation strategies are proposed—robust prompts, automated bias detection, model calibration with domain-specific training, and ensemble/human-in-the-loop oversight—aimed at enabling fair, trustworthy AI evaluation in telecom and network management contexts.

Abstract

Large Language Models (LLMs) are increasingly being used to autonomously evaluate the quality of content in communication systems, e.g., to assess responses in telecom customer support chatbots. However, the impartiality of these AI "judges" is not guaranteed, and any biases in their evaluation criteria could skew outcomes and undermine user trust. In this paper, we systematically investigate judgment biases in two LLM-as-a-judge models (i.e., GPT-Judge and JudgeLM) under the point-wise scoring setting, encompassing 11 types of biases that cover both implicit and explicit forms. We observed that state-of-the-art LLM judges demonstrate robustness to biased inputs, generally assigning them lower scores than the corresponding clean samples. Providing a detailed scoring rubric further enhances this robustness. We further found that fine-tuning an LLM on high-scoring yet biased responses can significantly degrade its performance, highlighting the risk of training on biased data. We also discovered that the judged scores correlate with task difficulty: a challenging dataset like GPQA yields lower average scores, whereas an open-ended reasoning dataset (e.g., JudgeLM-val) sees higher average scores. Finally, we proposed four potential mitigation strategies to ensure fair and reliable AI judging in practical communication scenarios.
Paper Structure (13 sections, 2 figures, 3 tables)

This paper contains 13 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. LLM-as-a-Judge in Communication Systems
  4. Evaluation Bias in LLM-as-a-Judge
  5. Evaluation Bias Impact on LLM-as-a-judge
  6. Experimental Setup
  7. Performance Evaluation
  8. Potential Bias Mitigations
  9. Robust Prompt Design and Reasoning
  10. Bias Detection Mechanisms
  11. Model Calibration and Specialized Training
  12. Ensemble of Judges and Human Oversight
  13. Conclusion

Figures (2)

  • Figure 1: Overview of the LLM-as-a-Judge framework in communication systems, illustrating how AI assistants in network management and telecom support are automatically evaluated by an LLM-based judge. We test 11 different types of bias in the experiments.
  • Figure 2: Average judge scores for answers with 11 different injected biases.