Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CMT-Bench: Cricket Multi-Table Generation Benchmark for Probing Robustness in Large Language Models

Ritam Upadhyay, Naman Ahuja, Rishabh Baral, Aparna Garimella, Vivek Gupta

TL;DR

CMT-Bench introduces a long-context Text-to-Table benchmark built from live cricket commentary to probe how models reason over temporally evolving narratives. It targets three robustness dimensions—removing extractive summaries, testing long-context stability, and perturbing entity forms—while grounding evaluation in two evolving scorecard schemas and ground-truth validation. Ground-truth tables are generated via regex-based parsing and validated against official scorecards, ensuring reliable benchmarks. Across six frontier LLMs, results reveal strong reliance on summary cues, monotonic degradation with context length, and substantial distributional shifts under entity perturbations, underscoring a need for robustness-first evaluation and guiding future state-update and schema-aware approaches.

Abstract

LLM Driven text-to-table (T2T) systems often rely on extensive prompt-engineering or iterative event extraction in code-parsable formats, which boosts scores but are computationally expensive and obscure how models actually reason over temporal evolving narratives to summarise key information. We present CMT-Bench, a diagnostic benchmark built from live cricket commentary that requires dynamic table generation across two evolving schemas under a dense, rule-governed policy. CMT-Bench is designed to probe robustness via three semantics-preserving dimensions: (i) extractive-cue ablation to separate extractive shortcuts from state tracking, (ii) temporal prefixing to test long-context stability, and (iii) entity-form perturbations (anonymization, outof-distribution substitutions, role-entangling paraphrases) to assess sensitivity to surface variation. Across diverse long-context stateof-the-art LLMs, we find large drops without extractive summaries, monotonic degradation with input length, and consistent accuracy drop under entity-form changes. Complementary distributional tests confirm significant shifts in numeric error patterns, indicating drift in reasoning rather than mere noise. Our results show that current LLMs are brittle in dynamic Textto-table generation, motivating robustness-first evaluation as a prerequisite for developing efficient and scalable approaches for this task.

CMT-Bench: Cricket Multi-Table Generation Benchmark for Probing Robustness in Large Language Models

TL;DR

CMT-Bench introduces a long-context Text-to-Table benchmark built from live cricket commentary to probe how models reason over temporally evolving narratives. It targets three robustness dimensions—removing extractive summaries, testing long-context stability, and perturbing entity forms—while grounding evaluation in two evolving scorecard schemas and ground-truth validation. Ground-truth tables are generated via regex-based parsing and validated against official scorecards, ensuring reliable benchmarks. Across six frontier LLMs, results reveal strong reliance on summary cues, monotonic degradation with context length, and substantial distributional shifts under entity perturbations, underscoring a need for robustness-first evaluation and guiding future state-update and schema-aware approaches.

Abstract

LLM Driven text-to-table (T2T) systems often rely on extensive prompt-engineering or iterative event extraction in code-parsable formats, which boosts scores but are computationally expensive and obscure how models actually reason over temporal evolving narratives to summarise key information. We present CMT-Bench, a diagnostic benchmark built from live cricket commentary that requires dynamic table generation across two evolving schemas under a dense, rule-governed policy. CMT-Bench is designed to probe robustness via three semantics-preserving dimensions: (i) extractive-cue ablation to separate extractive shortcuts from state tracking, (ii) temporal prefixing to test long-context stability, and (iii) entity-form perturbations (anonymization, outof-distribution substitutions, role-entangling paraphrases) to assess sensitivity to surface variation. Across diverse long-context stateof-the-art LLMs, we find large drops without extractive summaries, monotonic degradation with input length, and consistent accuracy drop under entity-form changes. Complementary distributional tests confirm significant shifts in numeric error patterns, indicating drift in reasoning rather than mere noise. Our results show that current LLMs are brittle in dynamic Textto-table generation, motivating robustness-first evaluation as a prerequisite for developing efficient and scalable approaches for this task.
Paper Structure (38 sections, 6 equations, 5 figures, 13 tables)

This paper contains 38 sections, 6 equations, 5 figures, 13 tables.

Table of Contents

  1. Introduction
  2. CMT-Bench
  3. Dataset Collection and Preprocessing
  4. Ground Truth Creation
  5. Ground-Truth Validation
  6. Robustness Study Design
  7. RQ1: Extractive cues vs. reasoning
  8. RQ2: Temporal robustness (with/without summary cues)
  9. RQ3: Entity-form robustness
  10. Models
  11. Evaluation Metrics
  12. Results
  13. RQ1: Exraction Cues v/s Reasoning
  14. RQ2: Temporal Robustness
  15. RQ3: Entity Form Robustness
  16. ...and 23 more sections

Figures (5)

  • Figure 1: Ball by Ball Commentary in Cricket and updation of Batsman and bowler Tables with every ball
  • Figure 2: Temporal cell accuracy trends with summary cues for (a) batsman and (b) bowler tables.
  • Figure 3: Temporal Trend of Accuracy on Cricket Score Tables
  • Figure 4: Comparison of avg. distribution shifts across models (left) and perturbation types (right). Lower shift values indicate higher robustness.
  • Figure 5: Comparison of avg. distribution shifts across models (left) and perturbation types (right). Lower shift values indicate higher robustness.