Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Translating Across Cultures: LLMs for Intralingual Cultural Adaptation

Pushpdeep Singh, Mayur Patidar, Lovekesh Vig

TL;DR

The paper defines intralingual cultural adaptation as a freestanding translation task where source-English dialogs are adapted to target culturally relevant English text. It introduces a CSI-centered evaluation framework, annotates a Friends-dialog corpus to capture culture-bound references, and analyzes open-source LLMs (Llama-2 70B, Llama-3 8B, Llama-3 70B) on their ability to localize, preserve content, and maintain naturalness in cross-cultural adaptation. The study provides both edit-level and dialog-level metrics, reveals model-dependent strengths and trade-offs, and validates automatic evaluation against human judgments. Overall, the work demonstrates that LLMs can localize cultural content to a meaningful degree but often struggle with coherence and preserving the original intent, highlighting avenues for refining prompts and integrating domain knowledge for better cross-cultural generation. This framework and dataset offer a foundation for evaluating culturally aware natural language generation and guiding future cross-cultural adaptation research.

Abstract

LLMs are increasingly being deployed for multilingual applications and have demonstrated impressive translation capabilities between several low and high-resource languages. An aspect of translation that often gets overlooked is that of cultural adaptation, or modifying source culture references to suit the target culture. While specialized translation models still outperform LLMs on the machine translation task when viewed from the lens of correctness, they are not sensitive to cultural differences often requiring manual correction. LLMs on the other hand have a rich reservoir of cultural knowledge embedded within its parameters that can be potentially exploited for such applications. In this paper, we define the task of cultural adaptation and create an evaluation framework to evaluate the performance of modern LLMs for cultural adaptation and analyze their cross-cultural knowledge while connecting related concepts across different cultures. We also analyze possible issues with automatic adaptation. We hope that this task will offer more insight into the cultural understanding of LLMs and their creativity in cross-cultural scenarios.

Translating Across Cultures: LLMs for Intralingual Cultural Adaptation

TL;DR

The paper defines intralingual cultural adaptation as a freestanding translation task where source-English dialogs are adapted to target culturally relevant English text. It introduces a CSI-centered evaluation framework, annotates a Friends-dialog corpus to capture culture-bound references, and analyzes open-source LLMs (Llama-2 70B, Llama-3 8B, Llama-3 70B) on their ability to localize, preserve content, and maintain naturalness in cross-cultural adaptation. The study provides both edit-level and dialog-level metrics, reveals model-dependent strengths and trade-offs, and validates automatic evaluation against human judgments. Overall, the work demonstrates that LLMs can localize cultural content to a meaningful degree but often struggle with coherence and preserving the original intent, highlighting avenues for refining prompts and integrating domain knowledge for better cross-cultural generation. This framework and dataset offer a foundation for evaluating culturally aware natural language generation and guiding future cross-cultural adaptation research.

Abstract

LLMs are increasingly being deployed for multilingual applications and have demonstrated impressive translation capabilities between several low and high-resource languages. An aspect of translation that often gets overlooked is that of cultural adaptation, or modifying source culture references to suit the target culture. While specialized translation models still outperform LLMs on the machine translation task when viewed from the lens of correctness, they are not sensitive to cultural differences often requiring manual correction. LLMs on the other hand have a rich reservoir of cultural knowledge embedded within its parameters that can be potentially exploited for such applications. In this paper, we define the task of cultural adaptation and create an evaluation framework to evaluate the performance of modern LLMs for cultural adaptation and analyze their cross-cultural knowledge while connecting related concepts across different cultures. We also analyze possible issues with automatic adaptation. We hope that this task will offer more insight into the cultural understanding of LLMs and their creativity in cross-cultural scenarios.
Paper Structure (18 sections, 7 figures, 14 tables)

This paper contains 18 sections, 7 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task Definition
  4. Annotating Cultural References
  5. Evaluation of Cultural Adaptation
  6. Aspects of Evaluation
  7. Edit Level Analysis
  8. Dialog Level Analysis
  9. Prompting for Cultural Adaptation
  10. Results and Analysis
  11. Edit level Analysis
  12. Dialog level Analysis
  13. Conclusion
  14. Limitations
  15. Prompts for LLM Evaluation
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Cultural Adaptation using LLM
  • Figure 2: Newmark1988's V diagram of translation methods. SL: Source Language, TL: Target Language
  • Figure 3: Hall's Iceberg Theory and Triads
  • Figure 4: Number of Occurrences of CSI by a) Category, b) Foreignness level. A total of 3192 occurrences were found.
  • Figure 5: Percentage of CSI edited in a) total, b) along different categories and c) foreignness level.
  • ...and 2 more figures