Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis

Zhiyin Tan, Jennifer D'Souza

TL;DR

This work reframes automated evidence extraction for meta-analysis as a structural, schema-driven diagnostic task, revealing that current LLMs struggle to maintain correct relational bindings and numerical grounding across documents, especially under long-context and high-arity conditions. By evaluating a tiered set of schema-constrained queries across five domains with two state-of-the-art LLMs, the authors pinpoint failure modes such as role reversals, binding drift, and instance compression that propagate errors into downstream meta-analytic statistics. The study demonstrates that single-atom extractions are feasible, but reliable corpus-level synthesis requires robust cross-document binding and verification, underscoring the need for schema-aware, neural–symbolic or constraint-guided architectures. The authors provide a benchmark, data, and code to drive future work toward methods that preserve analytical scope, ensure consistent variable-role mapping, and enable trustworthy aggregation for evidence synthesis at scale.

Abstract

Systematic reviews and meta-analyses rely on converting narrative articles into structured, numerically grounded study records. Despite rapid advances in large language models (LLMs), it remains unclear whether they can meet the structural requirements of this process, which hinge on preserving roles, methods, and effect-size attribution across documents rather than on recognizing isolated entities. We propose a structural, diagnostic framework that evaluates LLM-based evidence extraction as a progression of schema-constrained queries with increasing relational and numerical complexity, enabling precise identification of failure points beyond atom-level extraction. Using a manually curated corpus spanning five scientific domains, together with a unified query suite and evaluation protocol, we evaluate two state-of-the-art LLMs under both per-document and long-context, multi-document input regimes. Across domains and models, performance remains moderate for single-property queries but degrades sharply once tasks require stable binding between variables, roles, statistical methods, and effect sizes. Full meta-analytic association tuples are extracted with near-zero reliability, and long-context inputs further exacerbate these failures. Downstream aggregation amplifies even minor upstream errors, rendering corpus-level statistics unreliable. Our analysis shows that these limitations stem not from entity recognition errors, but from systematic structural breakdowns, including role reversals, cross-analysis binding drift, instance compression in dense result sections, and numeric misattribution, indicating that current LLMs lack the structural fidelity, relational binding, and numerical grounding required for automated meta-analysis. The code and data are publicly available at GitHub (https://github.com/zhiyintan/LLM-Meta-Analysis).

Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis

TL;DR

This work reframes automated evidence extraction for meta-analysis as a structural, schema-driven diagnostic task, revealing that current LLMs struggle to maintain correct relational bindings and numerical grounding across documents, especially under long-context and high-arity conditions. By evaluating a tiered set of schema-constrained queries across five domains with two state-of-the-art LLMs, the authors pinpoint failure modes such as role reversals, binding drift, and instance compression that propagate errors into downstream meta-analytic statistics. The study demonstrates that single-atom extractions are feasible, but reliable corpus-level synthesis requires robust cross-document binding and verification, underscoring the need for schema-aware, neural–symbolic or constraint-guided architectures. The authors provide a benchmark, data, and code to drive future work toward methods that preserve analytical scope, ensure consistent variable-role mapping, and enable trustworthy aggregation for evidence synthesis at scale.

Abstract

Systematic reviews and meta-analyses rely on converting narrative articles into structured, numerically grounded study records. Despite rapid advances in large language models (LLMs), it remains unclear whether they can meet the structural requirements of this process, which hinge on preserving roles, methods, and effect-size attribution across documents rather than on recognizing isolated entities. We propose a structural, diagnostic framework that evaluates LLM-based evidence extraction as a progression of schema-constrained queries with increasing relational and numerical complexity, enabling precise identification of failure points beyond atom-level extraction. Using a manually curated corpus spanning five scientific domains, together with a unified query suite and evaluation protocol, we evaluate two state-of-the-art LLMs under both per-document and long-context, multi-document input regimes. Across domains and models, performance remains moderate for single-property queries but degrades sharply once tasks require stable binding between variables, roles, statistical methods, and effect sizes. Full meta-analytic association tuples are extracted with near-zero reliability, and long-context inputs further exacerbate these failures. Downstream aggregation amplifies even minor upstream errors, rendering corpus-level statistics unreliable. Our analysis shows that these limitations stem not from entity recognition errors, but from systematic structural breakdowns, including role reversals, cross-analysis binding drift, instance compression in dense result sections, and numeric misattribution, indicating that current LLMs lack the structural fidelity, relational binding, and numerical grounding required for automated meta-analysis. The code and data are publicly available at GitHub (https://github.com/zhiyintan/LLM-Meta-Analysis).
Paper Structure (18 sections, 3 equations, 2 figures, 6 tables)

This paper contains 18 sections, 3 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Modeling and Structured Representations of Scientific Studies.
  4. Automating Information Extraction in Systematic Reviews and Meta-Analyses.
  5. LLMs for Scientific Information Extraction and Multi-Document Reasoning.
  6. Compositionality, Binding, and Numeric Reasoning in LLMs.
  7. Neural--Symbolic and Schema-Aware Approaches to Evidence Synthesis.
  8. Methodology
  9. Semantic Study Schema and Application Scope
  10. Problem Formulation
  11. Evaluation Principles
  12. Experiments
  13. Corpora and Annotation
  14. Experimental Setup
  15. Results
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Partial view of the original ORKG template for Early Drought Stress Indicators in Plants (Physiological & Multi-Sensor)https://orkg.org/templates/R1471884, showing a richly nested, domain-specific schema (plant/specimen descriptors, multi-sensor indicators and instruments, environment/stress conditions, and structured results with value–range–unit measurements) that we generalize, in this study, into a cross-domain study schema for association-based evidence extraction.
  • Figure 2: Domain-level comparison of macro-averaged F1 scores under the Per-Paper and Global input regimes. Each line connects Per-Paper and Global performance for the same domain and model, illustrating the magnitude of performance change induced by input regime variation.