Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Lingual Stability and Bias in Instruction-Tuned Language Models for Humanitarian NLP

Poli Nemkova, Amrit Adhikari, Matthew Pearson, Vamsi Krishna Sadu, Mark V. Albert

TL;DR

This study tackles how instruction-tuned versus open-weight LLMs perform in multilingual human rights violation detection across seven languages. By analyzing 78,000 inferences using six models and novel cross-lingual reliability metrics (CD, $\Delta$Bias, LRS, LSS), the authors demonstrate that multilingual alignment drives language-agnostic reasoning, with alignment-bearing models maintaining stable accuracy and calibration across typologically diverse languages. In contrast, open-weight models exhibit strong prompt-language sensitivity and calibration drift, revealing significant cross-language reliability risks for resource-constrained deployments. The findings provide practical guidance for humanitarian efforts, advocating alignment-focused models and robustness diagnostics, and suggesting hybrid strategies when open-weight solutions are considered due to cost constraints.

Abstract

Humanitarian organizations face a critical choice: invest in costly commercial APIs or rely on free open-weight models for multilingual human rights monitoring. While commercial systems offer reliability, open-weight alternatives lack empirical validation -- especially for low-resource languages common in conflict zones. This paper presents the first systematic comparison of commercial and open-weight large language models (LLMs) for human-rights-violation detection across seven languages, quantifying the cost-reliability trade-off facing resource-constrained organizations. Across 78,000 multilingual inferences, we evaluate six models -- four instruction-aligned (Claude-Sonnet-4, DeepSeek-V3, Gemini-Flash-2.0, GPT-4.1-mini) and two open-weight (LLaMA-3-8B, Mistral-7B) -- using both standard classification metrics and new measures of cross-lingual reliability: Calibration Deviation (CD), Decision Bias (B), Language Robustness Score (LRS), and Language Stability Score (LSS). Results show that alignment, not scale, determines stability: aligned models maintain near-invariant accuracy and balanced calibration across typologically distant and low-resource languages (e.g., Lingala, Burmese), while open-weight models exhibit significant prompt-language sensitivity and calibration drift. These findings demonstrate that multilingual alignment enables language-agnostic reasoning and provide practical guidance for humanitarian organizations balancing budget constraints with reliability in multilingual deployment.

Cross-Lingual Stability and Bias in Instruction-Tuned Language Models for Humanitarian NLP

TL;DR

This study tackles how instruction-tuned versus open-weight LLMs perform in multilingual human rights violation detection across seven languages. By analyzing 78,000 inferences using six models and novel cross-lingual reliability metrics (CD, Bias, LRS, LSS), the authors demonstrate that multilingual alignment drives language-agnostic reasoning, with alignment-bearing models maintaining stable accuracy and calibration across typologically diverse languages. In contrast, open-weight models exhibit strong prompt-language sensitivity and calibration drift, revealing significant cross-language reliability risks for resource-constrained deployments. The findings provide practical guidance for humanitarian efforts, advocating alignment-focused models and robustness diagnostics, and suggesting hybrid strategies when open-weight solutions are considered due to cost constraints.

Abstract

Humanitarian organizations face a critical choice: invest in costly commercial APIs or rely on free open-weight models for multilingual human rights monitoring. While commercial systems offer reliability, open-weight alternatives lack empirical validation -- especially for low-resource languages common in conflict zones. This paper presents the first systematic comparison of commercial and open-weight large language models (LLMs) for human-rights-violation detection across seven languages, quantifying the cost-reliability trade-off facing resource-constrained organizations. Across 78,000 multilingual inferences, we evaluate six models -- four instruction-aligned (Claude-Sonnet-4, DeepSeek-V3, Gemini-Flash-2.0, GPT-4.1-mini) and two open-weight (LLaMA-3-8B, Mistral-7B) -- using both standard classification metrics and new measures of cross-lingual reliability: Calibration Deviation (CD), Decision Bias (B), Language Robustness Score (LRS), and Language Stability Score (LSS). Results show that alignment, not scale, determines stability: aligned models maintain near-invariant accuracy and balanced calibration across typologically distant and low-resource languages (e.g., Lingala, Burmese), while open-weight models exhibit significant prompt-language sensitivity and calibration drift. These findings demonstrate that multilingual alignment enables language-agnostic reasoning and provide practical guidance for humanitarian organizations balancing budget constraints with reliability in multilingual deployment.
Paper Structure (24 sections, 1 equation, 2 figures, 7 tables)

This paper contains 24 sections, 1 equation, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Cultural Alignment and Value Pluralism in LLMs.
  4. Sociopolitical and Demographic Biases.
  5. Multilingual Disparities and Prompt Sensitivity.
  6. Implications for Human Rights Risk Detection.
  7. Bias and Subgroup Error Analysis.
  8. Methods
  9. Datasets
  10. Models
  11. Experimental Design
  12. Language Conditions
  13. Implementation
  14. Evaluation Metrics
  15. Results and Discussion
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Experimental setup illustrating the use of two datasets, multiple multilingual prompts, and six large language models (LLMs).
  • Figure 2: Model–Prompt Language Interaction (Accuracy Heatmap). Each cell shows mean classification accuracy for a given model prompted in a specific language. Instruction-aligned models (Claude Sonnet 4, DeepSeek-V3, Gemini-Flash-2.0, GPT-4.1-mini) display consistently high accuracy across languages, indicating strong cross-lingual robustness, while open-weight models (LLaMA-3-8B-Instruct, Mistral-7B) exhibit pronounced prompt-language sensitivity and instability.