Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translations

Yimin Xiao, Yongle Zhang, Dayeon Ki, Calvin Bao, Marianna J. Martindale, Charlotte Vaughn, Ge Gao, Marine Carpuat

TL;DR

The paper addresses how lay users perceive and rely on imperfect machine translation in everyday contexts, highlighting the MT literacy gap. It deploys a museum-based study (n=452 valid) with a 2×3 design that manipulates translation correctness and error type, measuring translation quality perception, decision accuracy, confidence, and willingness to reuse MT across Spanish proficiency levels. Key findings show bilingual and non-bilingual users rely on MT differently; non-bilingual users often over-rely due to limited evaluation strategies, while error experiences can prompt reassessment of future use. The work advocates for MT evaluation tools and NLP explanations to support user understanding, suggesting practical directions for MT literacy interventions and interdisciplinary collaboration across NLP, HCI, and Translation Studies.

Abstract

As Machine Translation (MT) becomes increasingly commonplace, understanding how the general public perceives and relies on imperfect MT is crucial for contextualizing MT research in real-world applications. We present a human study conducted in a public museum (n=452), investigating how fluency and adequacy errors impact bilingual and non-bilingual users' reliance on MT during casual use. Our findings reveal that non-bilingual users often over-rely on MT due to a lack of evaluation strategies and alternatives, while experiencing the impact of errors can prompt users to reassess future reliance. This highlights the need for MT evaluation and NLP explanation techniques to promote not only MT quality, but also MT literacy among its users.

Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translations

TL;DR

The paper addresses how lay users perceive and rely on imperfect machine translation in everyday contexts, highlighting the MT literacy gap. It deploys a museum-based study (n=452 valid) with a 2×3 design that manipulates translation correctness and error type, measuring translation quality perception, decision accuracy, confidence, and willingness to reuse MT across Spanish proficiency levels. Key findings show bilingual and non-bilingual users rely on MT differently; non-bilingual users often over-rely due to limited evaluation strategies, while error experiences can prompt reassessment of future use. The work advocates for MT evaluation tools and NLP explanations to support user understanding, suggesting practical directions for MT literacy interventions and interdisciplinary collaboration across NLP, HCI, and Translation Studies.

Abstract

As Machine Translation (MT) becomes increasingly commonplace, understanding how the general public perceives and relies on imperfect MT is crucial for contextualizing MT research in real-world applications. We present a human study conducted in a public museum (n=452), investigating how fluency and adequacy errors impact bilingual and non-bilingual users' reliance on MT during casual use. Our findings reveal that non-bilingual users often over-rely on MT due to a lack of evaluation strategies and alternatives, while experiencing the impact of errors can prompt users to reassess future reliance. This highlights the need for MT evaluation and NLP explanation techniques to promote not only MT quality, but also MT literacy among its users.

Paper Structure

This paper contains 28 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Research Questions & Background
  3. How is MT Used?
  4. What Makes MT"Good"?
  5. How is MT Perceived?
  6. Research Questions.
  7. Methods
  8. Study Design
  9. Stimuli Collection
  10. Construction.
  11. Validation.
  12. Participants
  13. Dependent Variables
  14. Results
  15. RQs 1 & 2: Perception of MT and Decision Making Based on Translations
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Example stimulus (Stimulus) with two candidate images for participants to select from (Selection). Image A shows a small outdoor library box and image B shows a small, low wooden shelf holding several books.
  • Figure 2: Perception of Translation Quality (i.e., Believability) by MT Error Type (legend) and Spanish Proficiency (x-axis). Note: *p < .05; **p < .01; ***p < .001.
  • Figure 3: Decision Accuracy by MT Error Type and Spanish Proficiency. Note: *p < .05; **p < .01; ***p < .001.
  • Figure 4: Screenshot of the full task for an example stimulus. Left: Participants rate their perception of the translation quality. Right: Next, participants select one out of two images that best match the location described in the Spanish message and rate their confidence in their selection. Image A shows colorful text radiating in a circular shape. Image B shows two children looking at a colorful, abstract digital display on the wall.