Divine LLaMAs: Bias, Stereotypes, Stigmatization, and Emotion Representation of Religion in Large Language Models

TL;DR

The paper investigates how large language models represent religion through emotion attribution using persona-based prompts and the ISEAR dataset. By evaluating multiple models (Llama2, Llama3, Mistral, GPT-4o) across 19 religious personas and 7 emotions, it reveals substantial cross-model differences and biases, including strong refusal patterns for Judaism and Islam in some Llama models and relatively unbiased behavior in Mistral and GPT-4o. Key findings show Western-majority religions receive more nuanced portrayals, while Hinduism and Buddhism face stronger stereotypes, and sacred-emotion mappings are inconsistently applied, often aligned with observance level. The work highlights the need for diverse, representative training data and careful methodological design to avoid reinforcing cultural biases in AI systems and provides a framework for future exploration of religion and emotion in NLP.

Abstract

Emotions play important epistemological and cognitive roles in our lives, revealing our values and guiding our actions. Previous work has shown that LLMs display biases in emotion attribution along gender lines. However, unlike gender, which says little about our values, religion, as a socio-cultural system, prescribes a set of beliefs and values for its followers. Religions, therefore, cultivate certain emotions. Moreover, these rules are explicitly laid out and interpreted by religious leaders. Using emotion attribution, we explore how different religions are represented in LLMs. We find that: Major religions in the US and European countries are represented with more nuance, displaying a more shaded model of their beliefs. Eastern religions like Hinduism and Buddhism are strongly stereotyped. Judaism and Islam are stigmatized -- the models' refusal skyrocket. We ascribe these to cultural bias in LLMs and the scarcity of NLP literature on religion. In the rare instances where religion is discussed, it is often in the context of toxic language, perpetuating the perception of these religions as inherently toxic. This finding underscores the urgent need to address and rectify these biases. Our research underscores the crucial role emotions play in our lives and how our values influence them.

Figures (15)

• Figure 1: LLM (Llama3-8b) emotion attribution and generated explanations across different personas based on religious backgrounds (cultural Hindu, cultural Jew, cultural Catholic) for the event "When some friends betrayed my friendship" from the ISEAR dataset scherer1994evidence. The complete explanations are in Table \ref{['tab:app_expl1']} of Appendix \ref{['app:exp1']}.
• Figure 2: Refusal rate (%) by Llama2 models family (Llama2-7b, Llama2-13b and Llama2-70b) across religions. We differentiate between refusals and compliance: Refusal, Compliance.
• Figure 3: Refusal rate (%) by Llama3 models family and Mistral across religions. We differentiate between refusals and compliance: Refusal, Compliance.
• Figure 4: The 12 most frequent emotions attributed by Llama2 models family (Llama2-7b, Llama2-13b, Llama2-70b) to each religion. Emotions are aggregated across models. Religion levels: Devout, practicing, cultural, non-religious.
• Figure 5: The 12 most frequent emotions attributed by Llama3 models family (Llama3-8b, Llama3-70b) to each religion. Emotions are aggregated across models. Religion levels: Devout, practicing, cultural, non-religious.