Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SectEval: Evaluating the Latent Sectarian Preferences of Large Language Models

Aditya Maheshwari, Amit Gajkeshwar, Kaushal Sharma, Vivek Patel

Abstract

As Large Language Models (LLMs) becomes a popular source for religious knowledge, it is important to know if it treats different groups fairly. This study is the first to measure how LLMs handle the differences between the two main sects of Islam: Sunni and Shia. We present a test called SectEval, available in both English and Hindi, consisting of 88 questions, to check the bias-ness of 15 top LLM models, both proprietary and open-weights. Our results show a major inconsistency based on language. In English, many powerful models DeepSeek-v3 and GPT-4o often favored Shia answers. However, when asked the exact same questions in Hindi, these models switched to favoring Sunni answers. This means a user could get completely different religious advice just by changing languages. We also looked at how models react to location. Advanced models Claude-3.5 changed their answers to match the user's country-giving Shia answers to a user from Iran and Sunni answers to a user from Saudi Arabia. In contrast, smaller models (especially in Hindi) ignored the user's location and stuck to a Sunni viewpoint. These findings show that AI is not neutral; its religious ``truth'' changes depending on the language you speak and the country you claim to be from. The data set is available at https://github.com/secteval/SectEval/

SectEval: Evaluating the Latent Sectarian Preferences of Large Language Models

Abstract

As Large Language Models (LLMs) becomes a popular source for religious knowledge, it is important to know if it treats different groups fairly. This study is the first to measure how LLMs handle the differences between the two main sects of Islam: Sunni and Shia. We present a test called SectEval, available in both English and Hindi, consisting of 88 questions, to check the bias-ness of 15 top LLM models, both proprietary and open-weights. Our results show a major inconsistency based on language. In English, many powerful models DeepSeek-v3 and GPT-4o often favored Shia answers. However, when asked the exact same questions in Hindi, these models switched to favoring Sunni answers. This means a user could get completely different religious advice just by changing languages. We also looked at how models react to location. Advanced models Claude-3.5 changed their answers to match the user's country-giving Shia answers to a user from Iran and Sunni answers to a user from Saudi Arabia. In contrast, smaller models (especially in Hindi) ignored the user's location and stuck to a Sunni viewpoint. These findings show that AI is not neutral; its religious ``truth'' changes depending on the language you speak and the country you claim to be from. The data set is available at https://github.com/secteval/SectEval/
Paper Structure (23 sections, 4 figures, 4 tables)

This paper contains 23 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SectEval
  4. Dataset
  5. Expert Curation and Binary-Choice Formulation
  6. Experimental Setup and Model Selection
  7. Results
  8. Evaluation on English Language
  9. Evaluation on Hindi Language
  10. Evaluation on Impact of Regional Identity on Theological Adaptability
  11. Performance on Small, Medium, Large, Frontier & MoE Models
  12. Question topic-wise results
  13. Evaluation on Chain-of-Thought
  14. Statistical Analysis of Language-Induced Bias Shifts
  15. Conclusion
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Sectarian Theological Alignment of LLMs on Islamic Knowledge Questions (Hindi)
  • Figure 2: Sectarian Theological Alignment of LLMs on Islamic Knowledge Questions (English)
  • Figure 3: Model responses across regional contexts and languages
  • Figure 4: Major Branches of Islam and their Associated Schools of Thought. Source: Wikipedia, Islamic schools and branches.