Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection

Chenyang Huang, Abbas Ghaddar, Ivan Kobyzev, Mehdi Rezagholizadeh, Osmar R. Zaiane, Boxing Chen

TL;DR

OTTAWA tackles hallucination and omission detection in machine translation by reframing word alignment as a partial Optimal Transport problem augmented with an adaptive null vector to model unaligned words. The method computes forward and reverse transport maps under a one-sided constrained OT formulation, producing a final alignment that highlights missing correspondences while distinguishing between hallucinations and omissions. Evaluations on HalOmi across 18 language pairs show competitive performance with state-of-the-art internal and external baselines and demonstrate the ability to separate error types without access to MT internals. The approach also enables token-level and cross-lingual analysis, suggesting practical utility for diagnosing and improving MT systems, with future work targeting one-to-many alignments and broader language coverage.

Abstract

Recently, there has been considerable attention on detecting hallucinations and omissions in Machine Translation (MT) systems. The two dominant approaches to tackle this task involve analyzing the MT system's internal states or relying on the output of external tools, such as sentence similarity or MT quality estimators. In this work, we introduce OTTAWA, a novel Optimal Transport (OT)-based word aligner specifically designed to enhance the detection of hallucinations and omissions in MT systems. Our approach explicitly models the missing alignments by introducing a "null" vector, for which we propose a novel one-side constrained OT setting to allow an adaptive null alignment. Our approach yields competitive results compared to state-of-the-art methods across 18 language pairs on the HalOmi benchmark. In addition, it shows promising features, such as the ability to distinguish between both error types and perform word-level detection without accessing the MT system's internal states.

OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection

TL;DR

OTTAWA tackles hallucination and omission detection in machine translation by reframing word alignment as a partial Optimal Transport problem augmented with an adaptive null vector to model unaligned words. The method computes forward and reverse transport maps under a one-sided constrained OT formulation, producing a final alignment that highlights missing correspondences while distinguishing between hallucinations and omissions. Evaluations on HalOmi across 18 language pairs show competitive performance with state-of-the-art internal and external baselines and demonstrate the ability to separate error types without access to MT internals. The approach also enables token-level and cross-lingual analysis, suggesting practical utility for diagnosing and improving MT systems, with future work targeting one-to-many alignments and broader language coverage.

Abstract

Recently, there has been considerable attention on detecting hallucinations and omissions in Machine Translation (MT) systems. The two dominant approaches to tackle this task involve analyzing the MT system's internal states or relying on the output of external tools, such as sentence similarity or MT quality estimators. In this work, we introduce OTTAWA, a novel Optimal Transport (OT)-based word aligner specifically designed to enhance the detection of hallucinations and omissions in MT systems. Our approach explicitly models the missing alignments by introducing a "null" vector, for which we propose a novel one-side constrained OT setting to allow an adaptive null alignment. Our approach yields competitive results compared to state-of-the-art methods across 18 language pairs on the HalOmi benchmark. In addition, it shows promising features, such as the ability to distinguish between both error types and perform word-level detection without accessing the MT system's internal states.
Paper Structure (23 sections, 13 equations, 3 figures, 5 tables)

This paper contains 23 sections, 13 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Unsupervised Word Alignment
  5. Standard Optimal Transport
  6. Partial Optimal Transport
  7. OTTAWA
  8. Experimental Setup
  9. Datasets and Evaluation Metrics
  10. Baselines and Models
  11. Implementation Details
  12. Results
  13. Sentence-Level Detection
  14. Ablation Study
  15. Embedding
  16. ...and 8 more sections

Figures (3)

  • Figure 1: A hallucinatory German-English translation was detected by correctly identifying the null alignments with our Optimal Transport-based method. Here, if a target word is mapping to "NULL", it is likely to be hallucinated, as no source word supports its translation.
  • Figure 2: LaBSE cosine similarity score (y-axis) of samples, aggregated across 10 high-resource datasets, with either hallucination (red triangles) or omission (blue circles) as gold labels. The x-axis shows the sample count index.
  • Figure 3: Hallucination scores (x-axis) and Omission scores (y-axis) produced by our OTTAWA across the same samples used in Figure \ref{['fig:labase_plot']}. The dotted green line simply illustrates the diagonal.