Is MT Ready for the Next Crisis or Pandemic?
Vipasha Bansal, Elizabeth Brown, Chelsea Kendrick, Benjamin Pong, William D. Lewis
TL;DR
The study probes whether contemporary MT systems and LLMs can reliably support crisis communication for pandemic response by evaluating Google, Microsoft, GPT-4o, and Gemini on the TICO-19 dataset across 35 languages and two timeframes. It finds that pivot languages are largely usable, while many priority languages remain unreliable, particularly in EX directions and in Africa/Asia, with notable quality losses in some providers likely tied to domain shifts or data contamination. The work emphasizes usability over mere BLEU, highlights contamination risks, and demonstrates that near-complete language coverage does not guarantee field-ready translation in crisis contexts. The findings underscore the need for cautious deployment, human evaluation, and ongoing resource development to ensure MT/LLM tools meet critical crisis communication needs.
Abstract
Communication in times of crisis is essential. However, there is often a mismatch between the language of governments, aid providers, doctors, and those to whom they are providing aid. Commercial MT systems are reasonable tools to turn to in these scenarios. But how effective are these tools for translating to and from low resource languages, particularly in the crisis or medical domain? In this study, we evaluate four commercial MT systems using the TICO-19 dataset, which is composed of pandemic-related sentences from a large set of high priority languages spoken by communities most likely to be affected adversely in the next pandemic. We then assess the current degree of ``readiness'' for another pandemic (or epidemic) based on the usability of the output translations.