Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Models

Abhilasha Sancheti, Haozhe An, Rachel Rudinger

TL;DR

The contextualized embeddings of first names are examined and it is found that gender for Asian names is less discernible than non-Asian names, underlining the need to prioritize the development of inclusive and equitable technology.

Abstract

We study the presence of heteronormative biases and prejudice against interracial romantic relationships in large language models by performing controlled name-replacement experiments for the task of relationship prediction. We show that models are less likely to predict romantic relationships for (a) same-gender character pairs than different-gender pairs; and (b) intra/inter-racial character pairs involving Asian names as compared to Black, Hispanic, or White names. We examine the contextualized embeddings of first names and find that gender for Asian names is less discernible than non-Asian names. We discuss the social implications of our findings, underlining the need to prioritize the development of inclusive and equitable technology.

On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Models

TL;DR

The contextualized embeddings of first names are examined and it is found that gender for Asian names is less discernible than non-Asian names, underlining the need to prioritize the development of inclusive and equitable technology.

Abstract

We study the presence of heteronormative biases and prejudice against interracial romantic relationships in large language models by performing controlled name-replacement experiments for the task of relationship prediction. We show that models are less likely to predict romantic relationships for (a) same-gender character pairs than different-gender pairs; and (b) intra/inter-racial character pairs involving Asian names as compared to Black, Hispanic, or White names. We examine the contextualized embeddings of first names and find that gender for Asian names is less discernible than non-Asian names. We discuss the social implications of our findings, underlining the need to prioritize the development of inclusive and equitable technology.
Paper Structure (50 sections, 10 figures, 4 tables)

This paper contains 50 sections, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Models
  4. Dataset
  5. Prompt Selection
  6. Evaluation
  7. Studying the Influence of Gender Pairings
  8. Studying Intra/Inter-Racial Pairings
  9. Findings
  10. Same-gender relationships are less likely to be predicted as romantic than different-gender ones.
  11. The unnaturalness of movie scripts with name and gender substitutions could, in theory, provide an alternative explanation for the observed biases, but the evidence shows this is not the cause.
  12. Character pairs involving Asian names have lower romantic recall; however, we do not find strong evidence against interracial pairings.
  13. Analysis and Discussion
  14. Why does a model tend to predict fewer romantic relationships for racial pairings that involve Asian names?
  15. Does gender association have a stronger influence on model's prediction than race/ethnicity?
  16. ...and 35 more sections

Figures (10)

  • Figure 1: Sample conversation from DDRel jia2021ddrel dataset and relationships predicted by Llama2-7B when characters are replaced by names with and . LLM tends to predict differently despite the same conversation.
  • Figure 2: Recall of predicting romantic relationships from Llama2-7B for subset of the dataset where characters originally have different genders. Horizontal and vertical axes denote % female of the name replacing an originally female and male character name from the dialogue. The upper-triangle (lower-triangle) shows the scores when names are replaced preserving (swapping) the genders of characters' names as-is in the original conversation. We consider the names with lesser % female as male names for determining gender preservation for name-replacement.
  • Figure 3: Recall of predicting romantic relationships from Llama2-7B for subset of the dataset where characters have different genders and are replaced with names associated with different races/ethnicities.
  • Figure 4: Prompt template used in our experiments. "{char_a}", "{char_b}", and "{context}" are placeholders here and they are instantiated with character names and dialogues accordingly for model inference.
  • Figure 5: Precision, F1-score and Accuracy plots for romantic predictions from Llama2-7B model.
  • ...and 5 more figures