Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Examination Framework: A Task-Agnostic Diagnostic for Information Fidelity in Text-to-Text Generation

Tathagata Raha, Clement Christophe, Nada Saadi, Hamza A Javed, Marco AF Pimentel, Ronnie Rajan, Praveenkumar Kanithi

TL;DR

The paper addresses the gap in evaluating information fidelity for text-to-text generation by introducing the Cross-Examination Framework (CEF), a reference-free, multi-dimensional diagnostic that derives Coverage, Conformity, and Consistency through cross-examined, verifiable questions from source and output. It validates CEF across machine translation, abstractive summarization, and clinical note generation, demonstrating robust judge selection (DeepSeek-V3) and an optimal question count (N = 10) that balance reliability and efficiency, while exposing omissions and factual contradictions that traditional metrics miss. The study shows strong alignment between reference-free and with-reference CEF scores and correspondence with human judgments, and it introduces a round-trip mode to extend applicability to low-resource languages. Overall, CEF provides interpretable, semantic-fidelity diagnostics that complement traditional overlap and embedding metrics, offering practical utility for multi-task evaluation in diverse languages.

Abstract

Traditional metrics like BLEU and BERTScore fail to capture semantic fidelity in generative text-to-text tasks. We adapt the Cross-Examination Framework (CEF) for a reference-free, multi-dimensional evaluation by treating the source and candidate as independent knowledge bases. CEF generates verifiable questions from each text and performs a cross-examination to derive three interpretable scores: Coverage, Conformity, and Consistency. Validated across translation, summarization and clinical note-generation, our framework identifies critical errors, such as content omissions and factual contradictions, missed by standard metrics. A key contribution is a systematic robustness analysis to select a stable judge model. Crucially, the strong correlation between our reference-free and with-reference modes validates CEF's reliability without gold references. Furthermore, human expert validation demonstrates that CEF mismatching questions align with meaning-altering semantic errors higher than with non-semantic errors, particularly excelling at identifying entity-based and relational distortions.

Cross-Examination Framework: A Task-Agnostic Diagnostic for Information Fidelity in Text-to-Text Generation

TL;DR

The paper addresses the gap in evaluating information fidelity for text-to-text generation by introducing the Cross-Examination Framework (CEF), a reference-free, multi-dimensional diagnostic that derives Coverage, Conformity, and Consistency through cross-examined, verifiable questions from source and output. It validates CEF across machine translation, abstractive summarization, and clinical note generation, demonstrating robust judge selection (DeepSeek-V3) and an optimal question count (N = 10) that balance reliability and efficiency, while exposing omissions and factual contradictions that traditional metrics miss. The study shows strong alignment between reference-free and with-reference CEF scores and correspondence with human judgments, and it introduces a round-trip mode to extend applicability to low-resource languages. Overall, CEF provides interpretable, semantic-fidelity diagnostics that complement traditional overlap and embedding metrics, offering practical utility for multi-task evaluation in diverse languages.

Abstract

Traditional metrics like BLEU and BERTScore fail to capture semantic fidelity in generative text-to-text tasks. We adapt the Cross-Examination Framework (CEF) for a reference-free, multi-dimensional evaluation by treating the source and candidate as independent knowledge bases. CEF generates verifiable questions from each text and performs a cross-examination to derive three interpretable scores: Coverage, Conformity, and Consistency. Validated across translation, summarization and clinical note-generation, our framework identifies critical errors, such as content omissions and factual contradictions, missed by standard metrics. A key contribution is a systematic robustness analysis to select a stable judge model. Crucially, the strong correlation between our reference-free and with-reference modes validates CEF's reliability without gold references. Furthermore, human expert validation demonstrates that CEF mismatching questions align with meaning-altering semantic errors higher than with non-semantic errors, particularly excelling at identifying entity-based and relational distortions.
Paper Structure (22 sections, 3 equations, 4 figures, 11 tables)

This paper contains 22 sections, 3 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. CEF Reliability
  5. Experiments
  6. Datasets and models
  7. Robustness of the CEF Framework
  8. Evaluating Text-to-Text Quality
  9. Human validation
  10. Ablation on the number of questions
  11. Ablation Study: Impact of "YES-Only" Question Constraints
  12. Conclusion
  13. Limitations
  14. Appendix
  15. Experimental settings
  16. ...and 7 more sections

Figures (4)

  • Figure 1: (Top) Reference-free CEF evaluation using DeepSeek-V3 as judge across four languages, where NLLB-3.3B along with commercial systems (Google, Azure) show strong performance while performance decreases for Arabic and Japanese. (Bottom) Correlation between reference-free and with-reference CEF scores for Consistency and Coverage ($r=0.846, 0.845$), validating CEF’s reliability without gold references, while Conformity shows weaker correlation ($r=0.457$) due to lower variability.
  • Figure 2: Reference-free CEF performance across Clinical Note Generation and Abstractive Summarization tasks
  • Figure 3: CEF metrics (Coverage, Consistency) show stronger correlation with semantic metrics (BertScore), than surface metrics (BLEU/chrF) or Conformity, validating CEF’s unique focus on deep semantic fidelity beyond lexical overlap.
  • Figure 4: Comparison of generated versus re-evaluated answers for two prompting strategies: a Yes-only prompt (producing only “YES” answers) and a Mixed-answer prompt (allowing “YES”, “NO”, or “IDK”). Each confusion matrix illustrates how the initially generated answers align with re-evaluated answers when the same document is given as a context during re-evaluation. (left) YES-Only Prompt demonstrates high stability, while (right) Mixed-Answer Prompt shows significant instability for "NO" and "IDK" answers.