Who's Asking? Investigating Bias Through the Lens of Disability Framed Queries in LLMs

TL;DR

The paper addresses the risk that LLMs infer user demographics from prompt phrasing, with disability cues potentially biasing those inferences. It conducts a comprehensive audit across eight instruction-tuned LLMs (3B–72B) using the AccessEval framework, nine disability categories, six business domains, and five demographic attributes under neutral and disability-aware prompts. Key findings show models infer demographics in up to 97% of cases, disability contexts shift attribute distributions, and domain context can amplify biases, with larger models sometimes displaying greater sensitivity to disability cues. The study highlights persistent intersections between ableism and other stereotypes, proposes abstention calibration and counterfactual fine-tuning as remedies, and contributes an evaluation framework and data to encourage disability-inclusive benchmarking. This work underscores the need for robust fairness mechanisms in LLM alignment and has practical implications for privacy, accessibility, and equitable AI deployment.

Abstract

Large Language Models (LLMs) routinely infer users demographic traits from phrasing alone, which can result in biased responses, even when no explicit demographic information is provided. The role of disability cues in shaping these inferences remains largely uncharted. Thus, we present the first systematic audit of disability-conditioned demographic bias across eight state-of-the-art instruction-tuned LLMs ranging from 3B to 72B parameters. Using a balanced template corpus that pairs nine disability categories with six real-world business domains, we prompt each model to predict five demographic attributes - gender, socioeconomic status, education, cultural background, and locality - under both neutral and disability-aware conditions. Across a varied set of prompts, models deliver a definitive demographic guess in up to 97\% of cases, exposing a strong tendency to make arbitrary inferences with no clear justification. Disability context heavily shifts predicted attribute distributions, and domain context can further amplify these deviations. We observe that larger models are simultaneously more sensitive to disability cues and more prone to biased reasoning, indicating that scale alone does not mitigate stereotype amplification. Our findings reveal persistent intersections between ableism and other demographic stereotypes, pinpointing critical blind spots in current alignment strategies. We release our evaluation framework and results to encourage disability-inclusive benchmarking and recommend integrating abstention calibration and counterfactual fine-tuning to curb unwarranted demographic inference. Code and data will be released on acceptance.

Figures (5)

• Figure 1: Overall response rate of eight instruction-tuned LLMs averaged across attributes. Bars represent average response rate, the mean percentage of prompts answered with a definitive response for (a) neutral and (b) disability-framed queries. Colors group models by parameter scale.
• Figure 2: Attribute-wise response rate on neutral queries. Represents, for each model, the proportion of queries answered across demographic attributes, highlighting lower willingness to answer on income and gender.
• Figure 3: Response rate by disability category. Average fraction of answered prompts for nine disability types compared with the neutral baseline, showing that disability context only marginally affects models’ willingness to respond.
• Figure 4: Bias by disability type. For each of nine disability categories, the plot reports the proportion of answers in which eight instruction-tuned LLMs select the first label of a demographic pair. Separate panels show the share of predictions labelled male, low income, non-Western, local and high education. Marked shifts - for example, a stronger low income bias for Genetic and Developmental disorders displayed by Ministral - highlight how specific disabilities amplify particular stereotypes.
• Figure 5: Domain-conditioned disability bias. Using the same metric as Fig. \ref{['fig:rq3.1-attributes']}, scores are first averaged across the eight models and then plotted for every combination of six business domains (Education, Finance, Healthcare, Hospitality, Media, Technology) and the nine disability categories. In each panel, one curve per domain traces how the domain context modulates bias across disabilities for the corresponding demographic attribute (male, low income, non-Western, local, high education). Pronounced domain-specific spikes - e.g. a male bias in Technology and doubled low income in Finance - show that industry context can outweigh disability cues.