Framing Political Bias in Multilingual LLMs Across Pakistani Languages

TL;DR

This work addresses political bias in multilingual LLMs within Pakistan by introducing a culturally adapted Political Compass Test (PCT) coupled with layered framing analysis across Urdu, Punjabi, Sindhi, Pashto, and Balochi. The methodology maps model outputs to a two-dimensional ideological space spanned by $S_{eco}$ and $S_{soc}$ on a $[-10,10]$ scale, while also assessing framing through content, entities, and sentiment using Boydstun’s frame taxonomy. Evaluations of 13 SOTA LLMs across 11 socio-political topics reveal language-conditioned shifts, with English outputs tending toward liberal-left frames and regional languages showing authoritarian-left tendencies, plus model-specific bias patterns. The study also includes a large-scale framing analysis of 444,340 multilingual headlines and introduces a publicly released multilingual dataset to support reproducible, culturally grounded bias auditing in low-resource, non-Western contexts. Overall, the framework advances multilingual bias detection and highlights the need for language- and culture-aware safety and fairness measures in global NLP deployments.

Abstract

Large Language Models (LLMs) increasingly shape public discourse, yet most evaluations of political and economic bias have focused on high-resource, Western languages and contexts. This leaves critical blind spots in low-resource, multilingual regions such as Pakistan, where linguistic identity is closely tied to political, religious, and regional ideologies. We present a systematic evaluation of political bias in 13 state-of-the-art LLMs across five Pakistani languages: Urdu, Punjabi, Sindhi, Pashto, and Balochi. Our framework integrates a culturally adapted Political Compass Test (PCT) with multi-level framing analysis, capturing both ideological stance (economic/social axes) and stylistic framing (content, tone, emphasis). Prompts are aligned with 11 socio-political themes specific to the Pakistani context. Results show that while LLMs predominantly reflect liberal-left orientations consistent with Western training data, they exhibit more authoritarian framing in regional languages, highlighting language-conditioned ideological modulation. We also identify consistent model-specific bias patterns across languages. These findings show the need for culturally grounded, multilingual bias auditing frameworks in global NLP.

Figures (49)

• Figure 1: Illustrates political bias in multilingual LLMs using an Urdu response to PCT Statement 24, where culturally and religiously grounded language is misinterpreted as support for violence. When mapped along ideological (liberal-conservative) and topical (death penalty) axes, the response is flagged as political bias, highlighting how misinterpretation of Urdu content can induce misalignment and polarization.
• Figure 2: Overview of our proposed framework for political bias analysis for evaluating political bias in language models. The framework features a political compass approach for stance detection and decomposes bias into content and style dimensions, examining controversial topics across Pakistani languages.
• Figure 3: Kappa score heatmap illustrating Language Correctness (LC) and Statement Correctness (SC) for five regional languages across model-human agreement.
• Figure 4: Political leaning of open source and closed source models used for Pakistani language shows diverse inclination across LLM
• Figure 5: Entity-level analysis of Urdu language outputs across LLMs. The bar chart (left) shows entity prediction frequency for key institutions, while the right chart visualizes the top 10 entities associated with the “Religious and Minority Rights” topic, with circle sizes indicating mention frequency and colors representing sentiment reflecting entity prominence, highlighting model-specific focus and cultural alignment in politically sensitive contexts.