Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Corpus of Cross-lingual Dialogues with Minutes and Detection of Misunderstandings

Marko Čechovič, Natália Komorníková, Dominik Macháček, Ondřej Bojar

TL;DR

The paper introduces InCroMin, a realistic corpus of cross-lingual dialogues facilitated by automatic simultaneous speech translation, including minutes for cross-lingual summarization research across 12 languages. It provides a detailed account of data collection, post-processing, deidentification, and initial evaluation of misunderstandings, including manual annotations and an automatic detection study using the Gemini LLM. The authors quantify misunderstandings, analyze their sources (translation errors, delays, technical issues, genuine misunderstandings), and report preliminary Gemini performance (recall 77%, precision 47%). They also share user feedback on the dialogue translation tool, highlighting usability and UI-related challenges and suggesting directions for future improvements in dialogue management and translation quality.

Abstract

Speech processing and translation technology have the potential to facilitate meetings of individuals who do not share any common language. To evaluate automatic systems for such a task, a versatile and realistic evaluation corpus is needed. Therefore, we create and present a corpus of cross-lingual dialogues between individuals without a common language who were facilitated by automatic simultaneous speech translation. The corpus consists of 5 hours of speech recordings with ASR and gold transcripts in 12 original languages and automatic and corrected translations into English. For the purposes of research into cross-lingual summarization, our corpus also includes written summaries (minutes) of the meetings. Moreover, we propose automatic detection of misunderstandings. For an overview of this task and its complexity, we attempt to quantify misunderstandings in cross-lingual meetings. We annotate misunderstandings manually and also test the ability of current large language models to detect them automatically. The results show that the Gemini model is able to identify text spans with misunderstandings with recall of 77% and precision of 47%.

Corpus of Cross-lingual Dialogues with Minutes and Detection of Misunderstandings

TL;DR

The paper introduces InCroMin, a realistic corpus of cross-lingual dialogues facilitated by automatic simultaneous speech translation, including minutes for cross-lingual summarization research across 12 languages. It provides a detailed account of data collection, post-processing, deidentification, and initial evaluation of misunderstandings, including manual annotations and an automatic detection study using the Gemini LLM. The authors quantify misunderstandings, analyze their sources (translation errors, delays, technical issues, genuine misunderstandings), and report preliminary Gemini performance (recall 77%, precision 47%). They also share user feedback on the dialogue translation tool, highlighting usability and UI-related challenges and suggesting directions for future improvements in dialogue management and translation quality.

Abstract

Speech processing and translation technology have the potential to facilitate meetings of individuals who do not share any common language. To evaluate automatic systems for such a task, a versatile and realistic evaluation corpus is needed. Therefore, we create and present a corpus of cross-lingual dialogues between individuals without a common language who were facilitated by automatic simultaneous speech translation. The corpus consists of 5 hours of speech recordings with ASR and gold transcripts in 12 original languages and automatic and corrected translations into English. For the purposes of research into cross-lingual summarization, our corpus also includes written summaries (minutes) of the meetings. Moreover, we propose automatic detection of misunderstandings. For an overview of this task and its complexity, we attempt to quantify misunderstandings in cross-lingual meetings. We annotate misunderstandings manually and also test the ability of current large language models to detect them automatically. The results show that the Gemini model is able to identify text spans with misunderstandings with recall of 77% and precision of 47%.
Paper Structure (15 sections, 2 figures, 6 tables)

This paper contains 15 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. InCroMin Corpus
  3. Technical Setup
  4. Data Creation
  5. Post-processing
  6. Corpus Overview
  7. Detection of Misunderstandings
  8. Annotation Protocol
  9. Annotation Results
  10. Inter-Annotator Agreement
  11. Gemini Capabilities in Identifying Misunderstandings
  12. Users' Feedback on the Dialogue Translation Tool
  13. Conclusion
  14. Acknowledgments.
  15. Disclosure of Interests.

Figures (2)

  • Figure 1: Gemini 1.5 Pro system prompt for finding misunderstandings.
  • Figure 2: Subjectively assessed slowdown due to cross-lingual barrier (left) and level of tiredness compared to normal calls featuring one language (right).