Chained Prompting for Better Systematic Review Search Strategies

TL;DR

The paper addresses the need for high-recall, unbiased literature searches in evidence synthesis and introduces an LLM-based chained prompt engineering pipeline that mirrors manual SR search design by decomposing a review objective into PICO elements, mapping them to concepts, expanding keywords, and synthesizing a Boolean query. On the LEADSInstruct dataset, the approach achieves recall of $0.90$ under filtering and $0.87$ on full evaluation, outperforming baselines such as GPT-4o ($0.10$), LEADS ($0.24$), and LEADS+ensemble ($0.82$). Error analysis highlights objective specification and terminological alignment as critical factors for optimal retrieval. The work demonstrates that LLM-driven, transparent pipelines can scale evidence synthesis and practice, with potential extension to additional databases and broader SR workflows.

Abstract

Systematic reviews require the use of rigorously designed search strategies to ensure both comprehensive retrieval and minimization of bias. Conventional manual approaches, although methodologically systematic, are resource-intensive and susceptible to subjectivity, whereas heuristic and automated techniques frequently under-perform in recall unless supplemented by extensive expert input. We introduce a Large Language Model (LLM)-based chained prompt engineering framework for the automated development of search strategies in systematic reviews. The framework replicates the procedural structure of manual search design while leveraging LLMs to decompose review objectives, extract and formalize PICO elements, generate conceptual representations, expand terminologies, and synthesize Boolean queries. In addition to query construction, the framework exhibits superior performance in generating well-structured PICO elements relative to existing methods, thereby strengthening the foundation for high-recall search strategies. Evaluation on a subset of the LEADSInstruct dataset demonstrates that the framework attains a 0.9 average recall. These results significantly exceed the performance of existing approaches. Error analysis further highlights the critical role of precise objective specification and terminological alignment in optimizing retrieval effectiveness. These findings confirm the capacity of LLM-based pipelines to yield transparent, reproducible, and high-performing search strategies, and highlight their potential as scalable instruments for supporting evidence synthesis and evidence-based practice.

Figures (7)

• Figure 1: Overview of the proposed chain prompting model for systematic review search strategy generation
• Figure 2: PICO elements prompting diagram
• Figure 3: Concepts prompting diagram
• Figure 4: Keywords prompting diagram
• Figure 5: Query construction prompting diagram