Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MALT: Mechanistic Ablation of Lossy Translation in LLMs for a Low-Resource Language: Urdu

Taaha Saleem Bajwa

TL;DR

The paper addresses the challenge that LLMs underperform on low-resource languages like Urdu due to English-centric latent representations and lossy final-layer translations. It proposes MALT, a mechanistic ablation of translation features in the final layer combined with a dedicated Urdu MT to translate the latent English responses, preserving input cultural nuances. On two small LLMs, MALT yields substantial gains in Urdu response quality, with Llama-3.2-3b improving from $11.6\%$ to $55\%$ and Gemma-2-2b from $0\%$ to $15.6\%$ in evaluation, indicating the approach can significantly enhance multilingual performance for low-resource languages. The work highlights a path toward fairer AI accessibility for non-English speakers, while acknowledging limitations in generalizing to larger models and requiring further validation of cultural nuance preservation and ethical implications.

Abstract

LLMs are predominantly trained on English data, which leads to a significant drop in performance on low-resource languages. Understanding how LLMs handle these languages is crucial for improving their effectiveness. This study focuses on Urdu as a use case for exploring the challenges faced by LLMs in processing low-resource languages. LLMs primarily reason in English when prompted in another language, with the final layers acting as translators to convert the English response into the target language. This study finds that even for low-resource languages, the internal latent response of LLMs in English is quite coherent; however, the translation features are lossy and result in poor translations, leading to reduced performance. By mechanistically removing these translation features and using a separate translation model to translate the internal latent response of LLM, the performance of LLMs improves significantly while also preserving the cultural nuances of the input in low-resource languages.

MALT: Mechanistic Ablation of Lossy Translation in LLMs for a Low-Resource Language: Urdu

TL;DR

The paper addresses the challenge that LLMs underperform on low-resource languages like Urdu due to English-centric latent representations and lossy final-layer translations. It proposes MALT, a mechanistic ablation of translation features in the final layer combined with a dedicated Urdu MT to translate the latent English responses, preserving input cultural nuances. On two small LLMs, MALT yields substantial gains in Urdu response quality, with Llama-3.2-3b improving from to and Gemma-2-2b from to in evaluation, indicating the approach can significantly enhance multilingual performance for low-resource languages. The work highlights a path toward fairer AI accessibility for non-English speakers, while acknowledging limitations in generalizing to larger models and requiring further validation of cultural nuance preservation and ethical implications.

Abstract

LLMs are predominantly trained on English data, which leads to a significant drop in performance on low-resource languages. Understanding how LLMs handle these languages is crucial for improving their effectiveness. This study focuses on Urdu as a use case for exploring the challenges faced by LLMs in processing low-resource languages. LLMs primarily reason in English when prompted in another language, with the final layers acting as translators to convert the English response into the target language. This study finds that even for low-resource languages, the internal latent response of LLMs in English is quite coherent; however, the translation features are lossy and result in poor translations, leading to reduced performance. By mechanistically removing these translation features and using a separate translation model to translate the internal latent response of LLM, the performance of LLMs improves significantly while also preserving the cultural nuances of the input in low-resource languages.

Paper Structure

This paper contains 20 sections, 4 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Dataset
  4. Models
  5. Finding Translation Direction Across Layers
  6. Removing Translation Direction
  7. Experimental Results
  8. Error types
  9. Evaluation
  10. Are cultural nuances preserved?
  11. Conclusion
  12. Acknowledgments
  13. Limitations
  14. Ethical Considerations
  15. Ethics Statement
  16. ...and 5 more sections

Figures (10)

  • Figure 1: (Above) Baseline LLM operation for low-resource languages, showing poor translation quality due to lossy translation features in the final layers. (Below) Proposed method, where lossy translation features are removed from the final layers and replaced with a dedicated machine translation model, resulting in high-quality responses. Note: The gray boxes are included solely to aid reader comprehension and are not part of the methodology.
  • Figure 2: Percentage of correct responses for Baseline and MALT in Gemma-2-2b and Llama-3.2-3b.
  • Figure 3: Non-relevant error observed in MALT for Gemma-2-2b: the word 'gene' is mistakenly interpreted as 'ghost' due to their similar structure in Urdu. Additionally, the response includes references to ghosts in the Quran, the holy book of Muslims, who form the majority of speakers of the input low resource language (Urdu). This indicates that cultural context of input language is preserved in MALT.
  • Figure 4: Fluency error seen in MALT for Gemma-2-2b
  • Figure 5: Repetition error seen in MALT for Gemma-2-2b
  • ...and 5 more figures