Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Regional Bias in Large Language Models

M P V S Gopinadh, Kappara Lakshmi Sindhu, Soma Sekhar Pandu Ranga Raju P, Yesaswini Swarna

TL;DR

This work addresses regional bias in large language models by introducing FAZE, a prompt-based, behavioral evaluation framework that quantifies unwarranted region-specific commitments under contextually neutral conditions. Applied to ten prominent LLMs with a dataset of 100 prompts (yielding 1,000 responses), FAZE produces a 10-point bias score that enables cross-model benchmarking. The results reveal substantial variation in geographic bias across models, from 9.5 (GPT-3.5) to 2.5 (Claude 3.5 Sonnet), indicating that regional bias is not solely a function of scale but is shaped by training data, alignment objectives, and post-training choices. The paper contributes a lightweight, reproducible methodology for assessing geographic fairness and motivates targeted debiasing to improve inclusive, globally representative AI outputs.

Abstract

This study investigates regional bias in large language models (LLMs), an emerging concern in AI fairness and global representation. We evaluate ten prominent LLMs: GPT-3.5, GPT-4o, Gemini 1.5 Flash, Gemini 1.0 Pro, Claude 3 Opus, Claude 3.5 Sonnet, Llama 3, Gemma 7B, Mistral 7B, and Vicuna-13B using a dataset of 100 carefully designed prompts that probe forced-choice decisions between regions under contextually neutral scenarios. We introduce FAZE, a prompt-based evaluation framework that measures regional bias on a 10-point scale, where higher scores indicate a stronger tendency to favor specific regions. Experimental results reveal substantial variation in bias levels across models, with GPT-3.5 exhibiting the highest bias score (9.5) and Claude 3.5 Sonnet scoring the lowest (2.5). These findings indicate that regional bias can meaningfully undermine the reliability, fairness, and inclusivity of LLM outputs in real-world, cross-cultural applications. This work contributes to AI fairness research by highlighting the importance of inclusive evaluation frameworks and systematic approaches for identifying and mitigating geographic biases in language models.

Regional Bias in Large Language Models

TL;DR

This work addresses regional bias in large language models by introducing FAZE, a prompt-based, behavioral evaluation framework that quantifies unwarranted region-specific commitments under contextually neutral conditions. Applied to ten prominent LLMs with a dataset of 100 prompts (yielding 1,000 responses), FAZE produces a 10-point bias score that enables cross-model benchmarking. The results reveal substantial variation in geographic bias across models, from 9.5 (GPT-3.5) to 2.5 (Claude 3.5 Sonnet), indicating that regional bias is not solely a function of scale but is shaped by training data, alignment objectives, and post-training choices. The paper contributes a lightweight, reproducible methodology for assessing geographic fairness and motivates targeted debiasing to improve inclusive, globally representative AI outputs.

Abstract

This study investigates regional bias in large language models (LLMs), an emerging concern in AI fairness and global representation. We evaluate ten prominent LLMs: GPT-3.5, GPT-4o, Gemini 1.5 Flash, Gemini 1.0 Pro, Claude 3 Opus, Claude 3.5 Sonnet, Llama 3, Gemma 7B, Mistral 7B, and Vicuna-13B using a dataset of 100 carefully designed prompts that probe forced-choice decisions between regions under contextually neutral scenarios. We introduce FAZE, a prompt-based evaluation framework that measures regional bias on a 10-point scale, where higher scores indicate a stronger tendency to favor specific regions. Experimental results reveal substantial variation in bias levels across models, with GPT-3.5 exhibiting the highest bias score (9.5) and Claude 3.5 Sonnet scoring the lowest (2.5). These findings indicate that regional bias can meaningfully undermine the reliability, fairness, and inclusivity of LLM outputs in real-world, cross-cultural applications. This work contributes to AI fairness research by highlighting the importance of inclusive evaluation frameworks and systematic approaches for identifying and mitigating geographic biases in language models.
Paper Structure (12 sections, 1 equation, 1 figure, 2 tables)

This paper contains 12 sections, 1 equation, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Methodology
  4. Framework Design
  5. Prompt Dataset Construction
  6. Model Selection
  7. Implementation
  8. Response Classification and Bias Score Calculation
  9. Results
  10. Discussion
  11. Challenges and Limitations
  12. Conclusion

Figures (1)

  • Figure 1: FAZE scores across evaluated models