Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EvasionBench: Detecting Evasive Answers in Financial Q&A via Multi-Model Consensus and LLM-as-Judge

Shijian Ma, Yan Lin, Yi Yang

TL;DR

EvasionBench addresses the lack of large-scale benchmarks for detecting evasive answers in earnings calls by introducing a 30k training and 1k test dataset assembled via a disagreement-driven, multi-model annotation framework with an LLM-as-Judge. The approach mines boundary cases where frontier models disagree and uses a judge to produce robust, scalable training labels, yielding improved generalization over single-teacher distillation. A 4B-parameter model, Eva-4B, achieves 81.3% accuracy, outperforming its base by 25.1 percentage points and approaching frontier LLM performance at a fraction of inference cost. The work demonstrates that disagreement-based hard-sample mining acts as effective regularization and provides a practical resource for financial NLP research and tools aimed at enhancing earnings-call transparency.

Abstract

Detecting evasive answers in earnings calls is critical for financial transparency, yet progress is hindered by the lack of large-scale benchmarks. We introduce EvasionBench, comprising 30,000 training samples and 1,000 human-annotated test samples (Cohen's Kappa 0.835) across three evasion levels. Our key contribution is a multi-model annotation framework leveraging a core insight: disagreement between frontier LLMs signals hard examples most valuable for training. We mine boundary cases where two strong annotators conflict, using a judge to resolve labels. This approach outperforms single-model distillation by 2.4 percent, with judge-resolved samples improving generalization despite higher training loss (0.421 vs 0.393) - evidence that disagreement mining acts as implicit regularization. Our trained model Eva-4B (4B parameters) achieves 81.3 percent accuracy, outperforming its base by 25 percentage points and approaching frontier LLM performance at a fraction of inference cost.

EvasionBench: Detecting Evasive Answers in Financial Q&A via Multi-Model Consensus and LLM-as-Judge

TL;DR

EvasionBench addresses the lack of large-scale benchmarks for detecting evasive answers in earnings calls by introducing a 30k training and 1k test dataset assembled via a disagreement-driven, multi-model annotation framework with an LLM-as-Judge. The approach mines boundary cases where frontier models disagree and uses a judge to produce robust, scalable training labels, yielding improved generalization over single-teacher distillation. A 4B-parameter model, Eva-4B, achieves 81.3% accuracy, outperforming its base by 25.1 percentage points and approaching frontier LLM performance at a fraction of inference cost. The work demonstrates that disagreement-based hard-sample mining acts as effective regularization and provides a practical resource for financial NLP research and tools aimed at enhancing earnings-call transparency.

Abstract

Detecting evasive answers in earnings calls is critical for financial transparency, yet progress is hindered by the lack of large-scale benchmarks. We introduce EvasionBench, comprising 30,000 training samples and 1,000 human-annotated test samples (Cohen's Kappa 0.835) across three evasion levels. Our key contribution is a multi-model annotation framework leveraging a core insight: disagreement between frontier LLMs signals hard examples most valuable for training. We mine boundary cases where two strong annotators conflict, using a judge to resolve labels. This approach outperforms single-model distillation by 2.4 percent, with judge-resolved samples improving generalization despite higher training loss (0.421 vs 0.393) - evidence that disagreement mining acts as implicit regularization. Our trained model Eva-4B (4B parameters) achieves 81.3 percent accuracy, outperforming its base by 25 percentage points and approaching frontier LLM performance at a fraction of inference cost.
Paper Structure (44 sections, 3 figures, 4 tables)

This paper contains 44 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Evasion Detection and Financial NLP
  5. Knowledge Distillation and Data Annotation
  6. LLM-as-Judge and Constitutional AI
  7. Hard Sample Mining and Active Learning
  8. Task Definition
  9. EvasionBench Dataset
  10. Data Collection and Source Data Preparation
  11. Stage 1: Q&A Pair Extraction
  12. Stage 2: Quality Filtering
  13. Stage 3: Substantial Content Selection
  14. Source Corpus Construction
  15. Multi-Model Annotation Framework
  16. ...and 29 more sections

Figures (3)

  • Figure 1: Overview of our multi-model annotation framework. Claude Opus 4.5 and Gemini-3-Flash independently annotate samples, with disagreements resolved by Claude Opus 4.5 as judge.
  • Figure 2: Task examples illustrating three evasion levels in earnings call Q&A.
  • Figure 3: Training loss curves for single-model baseline versus Eva-4B. Lower training loss does not guarantee better test performance, demonstrating that judge-resolved samples act as regularization.