Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMs

Mahammed Kamruzzaman, Gene Louis Kim

Abstract

Persona assignment has become a common strategy for customizing LLM use to particular tasks and contexts. In this study, we explore how evaluation of different nations change when LLMs are assigned specific nationality personas. We assign 193 different nationality personas (e.g., an American person) to four LLMs and examine how the LLM evaluations (or ''perceptions'')of countries change. We find that all LLM-persona combinations tend to favor Western European nations, though nation-personas push LLM behaviors to focus more on and treat the nation-persona's own region more favorably. Eastern European, Latin American, and African nations are treated more negatively by different nationality personas. We additionally find that evaluations by nation-persona LLMs of other nations correlate with human survey responses but fail to match the values closely. Our study provides insight into how biases and stereotypes are realized within LLMs when adopting different national personas. In line with the ''Blueprint for an AI Bill of Rights'', our findings underscore the critical need for developing mechanisms to ensure that LLM outputs promote fairness and avoid over-generalization.

Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMs

Abstract

Persona assignment has become a common strategy for customizing LLM use to particular tasks and contexts. In this study, we explore how evaluation of different nations change when LLMs are assigned specific nationality personas. We assign 193 different nationality personas (e.g., an American person) to four LLMs and examine how the LLM evaluations (or ''perceptions'')of countries change. We find that all LLM-persona combinations tend to favor Western European nations, though nation-personas push LLM behaviors to focus more on and treat the nation-persona's own region more favorably. Eastern European, Latin American, and African nations are treated more negatively by different nationality personas. We additionally find that evaluations by nation-persona LLMs of other nations correlate with human survey responses but fail to match the values closely. Our study provides insight into how biases and stereotypes are realized within LLMs when adopting different national personas. In line with the ''Blueprint for an AI Bill of Rights'', our findings underscore the critical need for developing mechanisms to ensure that LLM outputs promote fairness and avoid over-generalization.
Paper Structure (37 sections, 7 figures, 12 tables)

This paper contains 37 sections, 7 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Creation
  4. Experimental Setup
  5. Models.
  6. Assigning personas and prompting templates.
  7. Metrics.
  8. Results and Discussion
  9. General Vs Nationality-assigned Persona
  10. Nationality-assigned personas lead to more diverse responses when compared to the general persona.
  11. Llama-3.1 shows the least variation in outputs based on nationality personas and Mistral-7B the most.
  12. Model and Region Level Analysis
  13. All models exhibit a pro-Western bias in terms of RP and PMR.
  14. RP and PMR values indicate a Western perspective.
  15. GPT-4o's behavior notably differs from the other LLMs.
  16. ...and 22 more sections

Figures (7)

  • Figure 1: International evaluative generation of American-persona-assigned GPT-4o.
  • Figure 2: World Map of Polarity Differences: This map shows the difference in positive and negative mentions for each country---where green is positive and red is negative.
  • Figure 3: General Persona Vs Nationality-Assigned Personas' Generation
  • Figure 4: RP values representation averaged across all the models.
  • Figure 5: PMR values representation averaged across all the models.
  • ...and 2 more figures