Transforming Evidence Synthesis: A Systematic Review of the Evolution of Automated Meta-Analysis in the Age of AI

TL;DR

This systematic review synthesizes the current state of Automated Meta-analysis (AMA) across medical and non-medical domains using a Progressive Phase Structure (PPS) framework and a Task-Technology Fit (TTF) lens. It finds strong gains in automating data pre-processing and information extraction, as well as some automated statistical modelling, but shows a substantial gap in end-to-end automation, particularly for advanced synthesis and bias assessment. The study identifies domain-specific patterns, highlights barriers to full automation, and outlines a roadmap emphasizing multi-stage integration, LLM fine-tuning, interpretability, living AMA, and multidisciplinary collaboration. By articulating concrete tool-by-task mappings and success/failure cases, the work provides practical guidance for advancing AMA toward scalable, domain-agnostic evidence synthesis with rigorous methodological safeguards.

Abstract

Exponential growth in scientific literature has heightened the demand for efficient evidence-based synthesis, driving the rise of the field of Automated Meta-analysis (AMA) powered by natural language processing and machine learning. This PRISMA systematic review introduces a structured framework for assessing the current state of AMA, based on screening 978 papers from 2006 to 2024, and analyzing 54 studies across diverse domains. Findings reveal a predominant focus on automating data processing (57%), such as extraction and statistical modeling, while only 17% address advanced synthesis stages. Just one study (2%) explored preliminary full-process automation, highlighting a critical gap that limits AMA's capacity for comprehensive synthesis. Despite recent breakthroughs in large language models (LLMs) and advanced AI, their integration into statistical modeling and higher-order synthesis, such as heterogeneity assessment and bias evaluation, remains underdeveloped. This has constrained AMA's potential for fully autonomous meta-analysis. From our dataset spanning medical (67%) and non-medical (33%) applications, we found that AMA has exhibited distinct implementation patterns and varying degrees of effectiveness in actually improving efficiency, scalability, and reproducibility. While automation has enhanced specific meta-analytic tasks, achieving seamless, end-to-end automation remains an open challenge. As AI systems advance in reasoning and contextual understanding, addressing these gaps is now imperative. Future efforts must focus on bridging automation across all meta-analysis stages, refining interpretability, and ensuring methodological robustness to fully realize AMA's potential for scalable, domain-agnostic synthesis.

Figures (6)

• Figure 1: PRISMA workflow
• Figure 2: Holistic framework for this review
• Figure 3: Progressive Phase Structure with TTF model
• Figure 4: Temporal patterns in AMA publications (A) and proportional discrepancies across different stages (B)
• Figure 5: Interdisciplinary applications of AMA across various domains