Input Matters: Evaluating Input Structure's Impact on LLM Summaries of Sports Play-by-Play

TL;DR

This work addresses the challenge of factual inaccuracies in LLM-generated sports summaries by systematically assessing how input structure affects fidelity. By comparing unstructured, row-structured, and JSON representations across two open-source LLMs and using a detailed manual error taxonomy, the study demonstrates that structured inputs—especially JSON—substantially reduce errors, with error rates dropping up to 69% for Llama and 65% for Qwen relative to unstructured data. A two-way repeated-measures ANOVA shows input structure explains over 80% of the variance in error rates and reveals significant interactions with model choice, underscoring that the benefits of structure are robust but model-dependent. The findings have practical implications for deploying accuracy-critical LLM systems in sports journalism and beyond, suggesting that careful input design can markedly improve factual fidelity, while outlining future work on automatic evaluation and cross-sport generalization.

Abstract

A major concern when deploying LLMs in accuracy-critical domains such as sports reporting is that the generated text may not faithfully reflect the input data. We quantify how input structure affects hallucinations and other factual errors in LLM-generated summaries of NBA play-by-play data, across three formats: row-structured, JSON and unstructured. We manually annotated 3,312 factual errors across 180 game summaries produced by two models, Llama-3.1-70B and Qwen2.5-72B. Input structure has a strong effect: JSON input reduces error rates by 69% for Llama and 65% for Qwen compared to unstructured input, while row-structured input reduces errors by 54% for Llama and 51% for Qwen. A two-way repeated measures ANOVA shows that input structure accounts for over 80% of the variance in error rates, with Tukey HSD post hoc tests confirming statistically significant differences between all input formats.

Figures (7)

• Figure 1: Hierarchical JSON representation of a single play-by-play event.(i) Event: LeBron James makes a three-point jump shot from 25 ft; (ii) Time: period (first overtime) and game clock (i.e., time remaining in that period); (iii) Score: points scored from the play and the teams' cumulative scores.
• Figure 2: Games by final margin. For each stratum (absolute score difference: $\leq 3$, 4--9, 10--19, $\geq 20$), the left bar shows the total number of NBA games in Dec. 2024 -- Jan. 2025, and the right bar shows the stratified sample used in our study. Values above bars are counts.
• Figure 3: Average error rates by input structure for two models (errors per 100 words).
• Figure 4: Average error rates (errors per 100 words) by error type and input structure. Each panel reports average error rates across the 30 summaries for that input format (Row, JSON, Unstructured), comparing Llama-3.1-70B and Qwen2.5-72B across seven error categories. Structured inputs (Row and JSON) yield significantly fewer total errors than unstructured inputs, with JSON providing the largest reduction.
• Figure 5: Examples of partial game summaries with error annotations, illustrating errors from the same game and period under the three input types, Row, JSON, and Unstructured (see \ref{['tab:row_example']}). Corrections are included for clarity. Notation: superscripts N, U, WO, WS, C, O, and X indicate the error categories: Name, Number, Word-objective, Word-subjective, Context, Other, and Not-checkable; this is used here for readability; see the Annotation Protocol \ref{['subsec: annotation']} for definitions.