Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Tasks

Yi-Long Lu, Chunhui Zhang, Wei Wang

TL;DR

The study shows that how a task asks for a response can systematically bias LLM judgments, with binary formats pushing toward negative or oppositional outcomes compared with continuous scales. It uses two tasks—value judgments and sentiment analysis—across multiple open-source and commercial LLMs, employing deterministic prompts and simulated human profiles, and analyzes the data with hierarchical Bayesian regression. The main finding is a robust negative bias in binary responses (e.g., reduced probabilities of positive judgments), while continuous judgments better track human references, though label mappings can still induce biases. These results highlight the critical role of task design in LLM-supported decision making and suggest practical mitigations such as calibration, post hoc adjustments, and careful prompt design to improve reliability in sensitive domains.

Abstract

Large Language Models (LLMs) are increasingly used in tasks such as psychological text analysis and decision-making in automated workflows. However, their reliability remains a concern due to potential biases inherited from their training process. In this study, we examine how different response format: binary versus continuous, may systematically influence LLMs' judgments. In a value statement judgments task and a text sentiment analysis task, we prompted LLMs to simulate human responses and tested both formats across several models, including both open-source and commercial models. Our findings revealed a consistent negative bias: LLMs were more likely to deliver "negative" judgments in binary formats compared to continuous ones. Control experiments further revealed that this pattern holds across both tasks. Our results highlight the importance of considering response format when applying LLMs to decision tasks, as small changes in task design can introduce systematic biases.

Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Tasks

TL;DR

The study shows that how a task asks for a response can systematically bias LLM judgments, with binary formats pushing toward negative or oppositional outcomes compared with continuous scales. It uses two tasks—value judgments and sentiment analysis—across multiple open-source and commercial LLMs, employing deterministic prompts and simulated human profiles, and analyzes the data with hierarchical Bayesian regression. The main finding is a robust negative bias in binary responses (e.g., reduced probabilities of positive judgments), while continuous judgments better track human references, though label mappings can still induce biases. These results highlight the critical role of task design in LLM-supported decision making and suggest practical mitigations such as calibration, post hoc adjustments, and careful prompt design to improve reliability in sensitive domains.

Abstract

Large Language Models (LLMs) are increasingly used in tasks such as psychological text analysis and decision-making in automated workflows. However, their reliability remains a concern due to potential biases inherited from their training process. In this study, we examine how different response format: binary versus continuous, may systematically influence LLMs' judgments. In a value statement judgments task and a text sentiment analysis task, we prompted LLMs to simulate human responses and tested both formats across several models, including both open-source and commercial models. Our findings revealed a consistent negative bias: LLMs were more likely to deliver "negative" judgments in binary formats compared to continuous ones. Control experiments further revealed that this pattern holds across both tasks. Our results highlight the importance of considering response format when applying LLMs to decision tasks, as small changes in task design can introduce systematic biases.
Paper Structure (20 sections, 4 equations, 4 figures, 1 table)

This paper contains 20 sections, 4 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Model Settings
  4. Experiment 1: Value Judgment
  5. Stimuli and Design
  6. Control Experiments
  7. Experiment 2: Sentiment Analysis
  8. Stimuli and Design
  9. Control Experiments
  10. Behavior Analysis
  11. Measurement of Response Bias
  12. Hierarchical Bayesian Regression of Decision Bias
  13. Value Judgment
  14. Sentiment Analysis
  15. Model Fitting
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Behavior results of value judgment. (A) Judgment curve of continuous vs. binary responses. LLMs are plotted in different columns. Dark solid lines represents Support: Yes condition, while lighter solid and dashed lines represent the two control conditions, Support: 1 and Support: 0. (B) Proportion of the Support category. Points are jittered for visualization. Error bars represents 95% CI. Llama-3.3-70b is short for Llama-3.3-70b-instruct, and Qwen-2.5-72b stands for Qwen-2.5-72b-instruct.
  • Figure 2: Fitted response bias for LLMs. All models show a bias opposing the statements. Colored dots represent results for each LLM, while the black dot indicates the mean bias across all models. Error bars stands for 95% HDI.
  • Figure 3: Comparison of LLM's responses under different conditions and human judgments in the sentiment analysis tasks. (A) Proportion of Positive category in continuous and binary judgments. Horizontal lines represent human results. Error bar stands for 95% CI. (B) Relationship of human and LLMs continuous judgments. (C) Judgment curve of human judgments vs. LLM binary responses in different conditions, 'Baseline' (solid black line, K or L means Positive), 'Positive: Yes' (solid line, Yes means Positive), 'Positive: No' (dashed line, No means Positive).
  • Figure 4: Fitted response bias in sentiment analysis. (A) All models show a bias for negative responses. (B) A bias toward "No". Colored dots represent results for each LLM, while the black dot indicates the mean bias across all models. Error bars stands for 95% HDI.