Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Equal Access, Unequal Interaction: A Counterfactual Audit of LLM Fairness

Alireza Amiri-Margavi, Arshia Gharagozlou, Amin Gholami Davodi, Seyed Pouyan Mousavi Davoudi, Hamidreza Hasani Balyani

TL;DR

This paper argues that fairness in large language models cannot be judged solely by access metrics, as identical access can mask disparities in interaction quality. It introduces a counterfactual, paired-prompt framework to audit GPT-4 and LLaMA-3.1-70B on career-advice tasks across eight identity configurations (age, gender, nationality), measuring access via refusals and interaction quality via automated linguistic metrics. The findings show 100% access for all identities but model-specific differences in tone, hedging, and sentiment, underscoring the need for post-access fairness evaluations in high-stakes advisory contexts. The work provides a lightweight, reproducible methodology for post-deployment auditing that complements traditional refusal-based fairness analyses, with implications for trust and decision-making in real-world usage.

Abstract

Prior work on fairness in large language models (LLMs) has primarily focused on access-level behaviors such as refusals and safety filtering. However, equitable access does not ensure equitable interaction quality once a response is provided. In this paper, we conduct a controlled fairness audit examining how LLMs differ in tone, uncertainty, and linguistic framing across demographic identities after access is granted. Using a counterfactual prompt design, we evaluate GPT-4 and LLaMA-3.1-70B on career advice tasks while varying identity attributes along age, gender, and nationality. We assess access fairness through refusal analysis and measure interaction quality using automated linguistic metrics, including sentiment, politeness, and hedging. Identity-conditioned differences are evaluated using paired statistical tests. Both models exhibit zero refusal rates across all identities, indicating uniform access. Nevertheless, we observe systematic, model-specific disparities in interaction quality: GPT-4 expresses significantly higher hedging toward younger male users, while LLaMA exhibits broader sentiment variation across identity groups. These results show that fairness disparities can persist at the interaction level even when access is equal, motivating evaluation beyond refusal-based audits.

Equal Access, Unequal Interaction: A Counterfactual Audit of LLM Fairness

TL;DR

This paper argues that fairness in large language models cannot be judged solely by access metrics, as identical access can mask disparities in interaction quality. It introduces a counterfactual, paired-prompt framework to audit GPT-4 and LLaMA-3.1-70B on career-advice tasks across eight identity configurations (age, gender, nationality), measuring access via refusals and interaction quality via automated linguistic metrics. The findings show 100% access for all identities but model-specific differences in tone, hedging, and sentiment, underscoring the need for post-access fairness evaluations in high-stakes advisory contexts. The work provides a lightweight, reproducible methodology for post-deployment auditing that complements traditional refusal-based fairness analyses, with implications for trust and decision-making in real-world usage.

Abstract

Prior work on fairness in large language models (LLMs) has primarily focused on access-level behaviors such as refusals and safety filtering. However, equitable access does not ensure equitable interaction quality once a response is provided. In this paper, we conduct a controlled fairness audit examining how LLMs differ in tone, uncertainty, and linguistic framing across demographic identities after access is granted. Using a counterfactual prompt design, we evaluate GPT-4 and LLaMA-3.1-70B on career advice tasks while varying identity attributes along age, gender, and nationality. We assess access fairness through refusal analysis and measure interaction quality using automated linguistic metrics, including sentiment, politeness, and hedging. Identity-conditioned differences are evaluated using paired statistical tests. Both models exhibit zero refusal rates across all identities, indicating uniform access. Nevertheless, we observe systematic, model-specific disparities in interaction quality: GPT-4 expresses significantly higher hedging toward younger male users, while LLaMA exhibits broader sentiment variation across identity groups. These results show that fairness disparities can persist at the interaction level even when access is equal, motivating evaluation beyond refusal-based audits.
Paper Structure (20 sections, 2 equations, 2 figures, 4 tables)

This paper contains 20 sections, 2 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Fairness and Bias in NLP.
  4. Fairness in Large Language Models.
  5. Counterfactual and Paired Evaluation.
  6. Methodology
  7. Models and Tasks
  8. Model and decoding settings.
  9. Counterfactual Identity Design
  10. Output Standardization
  11. Metrics and Analysis
  12. Access Fairness.
  13. Interaction Quality.
  14. Metric definitions.
  15. Statistical Analysis.
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Counterfactual interaction-fairness evaluation pipeline. Prompts are paired with identity descriptors, standardized via an output contract, evaluated across models, and compared using tone metrics and paired tests. Paired comparisons are performed across counterfactual identities for each prompt.
  • Figure 2: Distribution of sentiment scores for career advice responses across identity groups (n=30 prompts per group). Solid boxes represent GPT-4, dashed boxes represent LLaMA. While all identities receive positive responses (sentiment $> 0.5$), GPT-4 shows greater within-group variability compared to LLaMA's tighter distributions.