Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System

TL;DR

Manalyzer addresses the bottleneck of end-to-end meta-analysis by deploying a multi-agent system that orchestrates literature search, screening, data extraction, and reporting through tool calls. It mitigates two major hallucination modes in LLM-based pipelines—screening misranking and erroneous data extraction—via a hybrid review workflow, hierarchical extraction with self-proving, and a feedback checker. The authors construct a large benchmark of 729 papers across three domains with multimodal data and over 10,000 data points to rigorously evaluate performance. Experimental results indicate that Manalyzer substantially outperforms LLM baselines on paper screening and data extraction, demonstrating the practical viability of MAS-based approaches for automated meta-analysis.

Abstract

Meta-analysis is a systematic research methodology that synthesizes data from multiple existing studies to derive comprehensive conclusions. This approach not only mitigates limitations inherent in individual studies but also facilitates novel discoveries through integrated data analysis. Traditional meta-analysis involves a complex multi-stage pipeline including literature retrieval, paper screening, and data extraction, which demands substantial human effort and time. However, while LLM-based methods can accelerate certain stages, they still face significant challenges, such as hallucinations in paper screening and data extraction. In this paper, we propose a multi-agent system, Manalyzer, which achieves end-to-end automated meta-analysis through tool calls. The hybrid review, hierarchical extraction, self-proving, and feedback checking strategies implemented in Manalyzer significantly alleviate these two hallucinations. To comprehensively evaluate the performance of meta-analysis, we construct a new benchmark comprising 729 papers across 3 domains, encompassing text, image, and table modalities, with over 10,000 data points. Extensive experiments demonstrate that Manalyzer achieves significant performance improvements over the LLM baseline in multi meta-analysis tasks. Project page: https://black-yt.github.io/meta-analysis-page/ .

Figures (17)

• Figure 1: Meta-analysis Comparison. a) Manual: time-consuming. b) LLM-based: limited to specific steps, fails to achieve end-to-end automation, prone to screening and extraction hallucinations. c) Manalyzer (ours): end-to-end automation, significantly reduced hallucinations via workflow design.
• Figure 2: Overview of Manalyzer. Manalyzer uses multi-agent collaboration with tools to automate the full meta-analysis workflow: search, download, parsing, data extraction, data analysis.
• Figure 3: Long Paper Review. Use the knapsack algorithm to address the issue of long papers exceeding the context window limit of LLMs.
• Figure 3: Classification skills of different models in screening papers.
• Figure 4: Distribution of Paper Scores under Different Review Strategies. Hybrid review strategy improves score diversity and selects more suitable papers.