Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reading between the Lines: Can LLMs Identify Cross-Cultural Communication Gaps?

Sougata Saha, Saurabh Kumar Pandey, Harshit Gupta, Monojit Choudhury

TL;DR

This paper investigates cross-cultural gaps in understanding Goodreads reviews by identifying Culture-Specific Items (CSIs) and evaluating GPT-4o as a cultural mediator. It combines a user study with human annotations and GPT-4o-assisted post-processing to extract CSIs and assess model performance. The findings show a substantial prevalence of CSIs (83% of reviews) and modest but equitable GPT-4o performance (recall ~0.65, precision ~0.49), indicating both the existence of cross-cultural readability barriers and the need for stronger cultural mediation tools. The work provides a publicly available dataset and demonstrates a concrete methodology for evaluating and improving AI-assisted cross-cultural communication across domains.

Abstract

In a rapidly globalizing and digital world, content such as book and product reviews created by people from diverse cultures are read and consumed by others from different corners of the world. In this paper, we investigate the extent and patterns of gaps in understandability of book reviews due to the presence of culturally-specific items and elements that might be alien to users from another culture. Our user-study on 57 book reviews from Goodreads reveal that 83\% of the reviews had at least one culture-specific difficult-to-understand element. We also evaluate the efficacy of GPT-4o in identifying such items, given the cultural background of the reader; the results are mixed, implying a significant scope for improvement. Our datasets are available here: https://github.com/sougata-ub/reading_between_lines

Reading between the Lines: Can LLMs Identify Cross-Cultural Communication Gaps?

TL;DR

This paper investigates cross-cultural gaps in understanding Goodreads reviews by identifying Culture-Specific Items (CSIs) and evaluating GPT-4o as a cultural mediator. It combines a user study with human annotations and GPT-4o-assisted post-processing to extract CSIs and assess model performance. The findings show a substantial prevalence of CSIs (83% of reviews) and modest but equitable GPT-4o performance (recall ~0.65, precision ~0.49), indicating both the existence of cross-cultural readability barriers and the need for stronger cultural mediation tools. The work provides a publicly available dataset and demonstrates a concrete methodology for evaluating and improving AI-assisted cross-cultural communication across domains.

Abstract

In a rapidly globalizing and digital world, content such as book and product reviews created by people from diverse cultures are read and consumed by others from different corners of the world. In this paper, we investigate the extent and patterns of gaps in understandability of book reviews due to the presence of culturally-specific items and elements that might be alien to users from another culture. Our user-study on 57 book reviews from Goodreads reveal that 83\% of the reviews had at least one culture-specific difficult-to-understand element. We also evaluate the efficacy of GPT-4o in identifying such items, given the cultural background of the reader; the results are mixed, implying a significant scope for improvement. Our datasets are available here: https://github.com/sougata-ub/reading_between_lines

Paper Structure

This paper contains 33 sections, 17 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Culture: A Primer
  3. User Study
  4. Dataset
  5. Method
  6. Pilot Studies
  7. Actual Study
  8. Post Processing
  9. LLM-assisted post processing
  10. Evaluation
  11. Analysis
  12. Benchmarking GPT-4o as a Cultural Assistant
  13. Experiment Setup
  14. Analysis
  15. Discussion
  16. ...and 18 more sections

Figures (17)

  • Figure 1: Participant location-wise difficulty in understanding book reviews by country of origin of the book.
  • Figure 2: Inter Annotator Agreement at Review Level and Span Level across Countries.
  • Figure 3: Top 15 Span Clusters identified by participants across countries. Corresponding span text in Table \ref{['tab:top_15_spans']}.
  • Figure 4: Overlap between Human-identified difficult spans and GPT-4o-identified CSIs
  • Figure 5: Overlap percentage of fiction and non-fiction spans across countries.
  • ...and 12 more figures