Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues

John Mendonça, Isabel Trancoso, Alon Lavie

TL;DR

GenResCoh provides a large-scale, multilingual dataset of generated responses with explanations to target turn-level coherence, addressing the need for open-source dialogue evaluation. ECoh, a family of lightweight, LoRA-finetuned evaluators based on Qwen1.5-Chat, achieves strong multilingual coherence detection that can match or exceed GPT-3.5-Turbo while delivering high-quality explanations. The approach demonstrates robust cross-language generalization, including unseen languages and external FED-turn annotations, and highlights practical advantages of open-source evaluation over closed models like GPT-4. Limitations include language coverage, potential generation biases, and dependence on synthetic data, suggesting avenues for expansion and more diverse baselines. Overall, this work offers scalable, accessible tools for multilingual dialogue coherence assessment with strong empirical results and clear paths for future enhancement.

Abstract

Despite being heralded as the new standard for dialogue evaluation, the closed-source nature of GPT-4 poses challenges for the community. Motivated by the need for lightweight, open source, and multilingual dialogue evaluators, this paper introduces GenResCoh (Generated Responses targeting Coherence). GenResCoh is a novel LLM generated dataset comprising over 130k negative and positive responses and accompanying explanations seeded from XDailyDialog and XPersona covering English, French, German, Italian, and Chinese. Leveraging GenResCoh, we propose ECoh (Evaluation of Coherence), a family of evaluators trained to assess response coherence across multiple languages. Experimental results demonstrate that ECoh achieves multilingual detection capabilities superior to the teacher model (GPT-3.5-Turbo) on GenResCoh, despite being based on a much smaller architecture. Furthermore, the explanations provided by ECoh closely align in terms of quality with those generated by the teacher model.

ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues

TL;DR

GenResCoh provides a large-scale, multilingual dataset of generated responses with explanations to target turn-level coherence, addressing the need for open-source dialogue evaluation. ECoh, a family of lightweight, LoRA-finetuned evaluators based on Qwen1.5-Chat, achieves strong multilingual coherence detection that can match or exceed GPT-3.5-Turbo while delivering high-quality explanations. The approach demonstrates robust cross-language generalization, including unseen languages and external FED-turn annotations, and highlights practical advantages of open-source evaluation over closed models like GPT-4. Limitations include language coverage, potential generation biases, and dependence on synthetic data, suggesting avenues for expansion and more diverse baselines. Overall, this work offers scalable, accessible tools for multilingual dialogue coherence assessment with strong empirical results and clear paths for future enhancement.

Abstract

Despite being heralded as the new standard for dialogue evaluation, the closed-source nature of GPT-4 poses challenges for the community. Motivated by the need for lightweight, open source, and multilingual dialogue evaluators, this paper introduces GenResCoh (Generated Responses targeting Coherence). GenResCoh is a novel LLM generated dataset comprising over 130k negative and positive responses and accompanying explanations seeded from XDailyDialog and XPersona covering English, French, German, Italian, and Chinese. Leveraging GenResCoh, we propose ECoh (Evaluation of Coherence), a family of evaluators trained to assess response coherence across multiple languages. Experimental results demonstrate that ECoh achieves multilingual detection capabilities superior to the teacher model (GPT-3.5-Turbo) on GenResCoh, despite being based on a much smaller architecture. Furthermore, the explanations provided by ECoh closely align in terms of quality with those generated by the teacher model.
Paper Structure (43 sections, 2 figures, 16 tables)

This paper contains 43 sections, 2 figures, 16 tables.

Table of Contents

  1. Introduction
  2. GenResCoh responses dataset
  3. Dataset Sources
  4. Generation
  5. Development set
  6. Test set
  7. Human validation
  8. Statistical Analysis
  9. ECoh
  10. Problem Formulation
  11. Experimental Setup
  12. Model Specification
  13. Baselines
  14. Main Results
  15. GPT-3.5 performance with 4B parameters
  16. ...and 28 more sections

Figures (2)

  • Figure 1: Confusion matrices for the best models of each family (ECoh-1.8B-en, GPT-3.5-Turbo and Qwen1.5-7B-Chat) on FED-turn.
  • Figure 2: GPT-4 response validation guidelines.