Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Wording Steers the Evaluation: Framing Bias in LLM judges

Yerin Hwang, Dongryeol Lee, Taegwan Kang, Minwoo Lee, Kyomin Jung

TL;DR

This study demonstrates that LLM-based evaluation is vulnerable to framing bias, showing that two semantically equivalent framings ($P$ vs $\\neg P$) can drive inconsistent judgments across four high-stakes tasks and 14 judge models. By introducing a framing protocol and three metrics (Inconsistency, Acquiescence Bias, and Task-Induced Bias), the authors reveal systematic, model-family–dependent directionality in framing effects and task-specific biases. The findings argue that current LLM evaluation pipelines can be systematically biased and urge framing-aware protocols to improve reliability and fairness in AI evaluation. The work highlights practical implications for deploying LLM evaluators in safety-critical domains and suggests avenues for reducing framing sensitivity, including cross-task stability analyses and multilingual robustness.

Abstract

Large language models (LLMs) are known to produce varying responses depending on prompt phrasing, indicating that subtle guidance in phrasing can steer their answers. However, the impact of this framing bias on LLM-based evaluation, where models are expected to make stable and impartial judgments, remains largely underexplored. Drawing inspiration from the framing effect in psychology, we systematically investigate how deliberate prompt framing skews model judgments across four high-stakes evaluation tasks. We design symmetric prompts using predicate-positive and predicate-negative constructions and demonstrate that such framing induces significant discrepancies in model outputs. Across 14 LLM judges, we observe clear susceptibility to framing, with model families showing distinct tendencies toward agreement or rejection. These findings suggest that framing bias is a structural property of current LLM-based evaluation systems, underscoring the need for framing-aware protocols.

When Wording Steers the Evaluation: Framing Bias in LLM judges

TL;DR

This study demonstrates that LLM-based evaluation is vulnerable to framing bias, showing that two semantically equivalent framings ( vs ) can drive inconsistent judgments across four high-stakes tasks and 14 judge models. By introducing a framing protocol and three metrics (Inconsistency, Acquiescence Bias, and Task-Induced Bias), the authors reveal systematic, model-family–dependent directionality in framing effects and task-specific biases. The findings argue that current LLM evaluation pipelines can be systematically biased and urge framing-aware protocols to improve reliability and fairness in AI evaluation. The work highlights practical implications for deploying LLM evaluators in safety-critical domains and suggests avenues for reducing framing sensitivity, including cross-task stability analyses and multilingual robustness.

Abstract

Large language models (LLMs) are known to produce varying responses depending on prompt phrasing, indicating that subtle guidance in phrasing can steer their answers. However, the impact of this framing bias on LLM-based evaluation, where models are expected to make stable and impartial judgments, remains largely underexplored. Drawing inspiration from the framing effect in psychology, we systematically investigate how deliberate prompt framing skews model judgments across four high-stakes evaluation tasks. We design symmetric prompts using predicate-positive and predicate-negative constructions and demonstrate that such framing induces significant discrepancies in model outputs. Across 14 LLM judges, we observe clear susceptibility to framing, with model families showing distinct tendencies toward agreement or rejection. These findings suggest that framing bias is a structural property of current LLM-based evaluation systems, underscoring the need for framing-aware protocols.
Paper Structure (22 sections, 9 equations, 6 figures, 2 tables)

This paper contains 22 sections, 9 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Framing bias in LLMs
  4. Framing Bias in LLM judges
  5. Framing Protocol
  6. Evaluation Tasks
  7. Metrics
  8. Experiments
  9. Experimental settings
  10. Results
  11. Conclusion
  12. Reproducibility Checklists
  13. Computing Resources
  14. Experimental Configuration of LLMs
  15. Metric Definitions
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Framing bias in LLM-based evaluation. Under both P and ¬P framings, the LLM judge answers “Yes,” resulting in a contradictory evaluation.
  • Figure 2: Task-induced bias (left) and average inconsistency (right) across four tasks.
  • Figure 3: Prompt for truthfulness evaluation.
  • Figure 4: Prompt for jailbreak detection evaluation.
  • Figure 5: Prompt for toxicity detection evaluation.
  • ...and 1 more figures