Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lost in Translation? A Comparative Study on the Cross-Lingual Transfer of Composite Harms

Vaibhav Shukla, Hardik Sharma, Adith N Reganti, Soham Wasmatkar, Bagesh Kumar, Vrijendra Singh

TL;DR

This work tackles the risk that safety alignment learned on English does not generalize to multilingual users. It introduces CompositeHarm, a translation-based benchmark that fuses AttaQ (syntactic adversarial prompts) and MMSafetyBench (semantic/contextual harms) and translates them into five Indic languages alongside English, enabling a controlled cross-lingual probe using lightweight models. The results show pronounced safety degradation in Indic languages, with adversarial syntax transfer being particularly brittle, while semantic-context transfers are more robust but still imperfect; evasive and guardrail failures further reveal gaps not captured by binary metrics. The study highlights the need for language-family-specific alignment and composite cross-lingual, cross-domain evaluation to build scalable, resource-aware, and safer multilingual LLM deployments, especially for edge devices.

Abstract

Most safety evaluations of large language models (LLMs) remain anchored in English. Translation is often used as a shortcut to probe multilingual behavior, but it rarely captures the full picture, especially when harmful intent or structure morphs across languages. Some types of harm survive translation almost intact, while others distort or disappear. To study this effect, we introduce CompositeHarm, a translation-based benchmark designed to examine how safety alignment holds up as both syntax and semantics shift. It combines two complementary English datasets, AttaQ, which targets structured adversarial attacks, and MMSafetyBench, which covers contextual, real-world harms, and extends them into six languages: English, Hindi, Assamese, Marathi, Kannada, and Gujarati. Using three large models, we find that attack success rates rise sharply in Indic languages, especially under adversarial syntax, while contextual harms transfer more moderately. To ensure scalability and energy efficiency, our study adopts lightweight inference strategies inspired by edge-AI design principles, reducing redundant evaluation passes while preserving cross-lingual fidelity. This design makes large-scale multilingual safety testing both computationally feasible and environmentally conscious. Overall, our results show that translated benchmarks are a necessary first step, but not a sufficient one, toward building grounded, resource-aware, language-adaptive safety systems.

Lost in Translation? A Comparative Study on the Cross-Lingual Transfer of Composite Harms

TL;DR

This work tackles the risk that safety alignment learned on English does not generalize to multilingual users. It introduces CompositeHarm, a translation-based benchmark that fuses AttaQ (syntactic adversarial prompts) and MMSafetyBench (semantic/contextual harms) and translates them into five Indic languages alongside English, enabling a controlled cross-lingual probe using lightweight models. The results show pronounced safety degradation in Indic languages, with adversarial syntax transfer being particularly brittle, while semantic-context transfers are more robust but still imperfect; evasive and guardrail failures further reveal gaps not captured by binary metrics. The study highlights the need for language-family-specific alignment and composite cross-lingual, cross-domain evaluation to build scalable, resource-aware, and safer multilingual LLM deployments, especially for edge devices.

Abstract

Most safety evaluations of large language models (LLMs) remain anchored in English. Translation is often used as a shortcut to probe multilingual behavior, but it rarely captures the full picture, especially when harmful intent or structure morphs across languages. Some types of harm survive translation almost intact, while others distort or disappear. To study this effect, we introduce CompositeHarm, a translation-based benchmark designed to examine how safety alignment holds up as both syntax and semantics shift. It combines two complementary English datasets, AttaQ, which targets structured adversarial attacks, and MMSafetyBench, which covers contextual, real-world harms, and extends them into six languages: English, Hindi, Assamese, Marathi, Kannada, and Gujarati. Using three large models, we find that attack success rates rise sharply in Indic languages, especially under adversarial syntax, while contextual harms transfer more moderately. To ensure scalability and energy efficiency, our study adopts lightweight inference strategies inspired by edge-AI design principles, reducing redundant evaluation passes while preserving cross-lingual fidelity. This design makes large-scale multilingual safety testing both computationally feasible and environmentally conscious. Overall, our results show that translated benchmarks are a necessary first step, but not a sufficient one, toward building grounded, resource-aware, language-adaptive safety systems.
Paper Structure (21 sections, 1 figure, 3 tables)

This paper contains 21 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multilingual LLM Safety
  4. Adversarial and Multimodal Red Teaming
  5. The CompositeHarm Benchmark
  6. Source Datasets
  7. Curation and Translation Process
  8. Dataset Statistics
  9. Experimental Setup
  10. Models
  11. Evaluation Protocol
  12. Metrics
  13. Evaluation Interpretation
  14. Results and Analysis
  15. Quantifying the Linguistic Distance Gap
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Cross-lingual safety results: model-wise RR and ASR across five languages and three models, highlighting variation in safety consistency and refusal behavior across languages.