Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification

TL;DR

The study investigates how prompting strategies interact with six large language models to automate the screening stage of systematic literature reviews. By constructing a ground-truth dataset of 1,376 studies (2014–2024) with detailed Level-1 and Level-2 annotations and evaluating five prompting strategies across six LLMs, the authors reveal pronounced model–prompt interactions and task-dependent performance. They find that chain-of-thought prompting with few-shot exemplars (CoT-few-shot) often yields the best precision–recall balance, while zero-shot prompting maximizes recall in high-sensitivity passes; self-reflection generally underperforms due to over-inclusivity and instability. A staged, cost-aware workflow is proposed: employ low-cost models with structured prompts for first-pass screening and escalate only borderline cases to higher-capacity models, supported by a comprehensive benchmark of model–prompt configurations and practical deployment guidance.

Abstract

This study quantifies how prompting strategies interact with large language models (LLMs) to automate the screening stage of systematic literature reviews (SLRs). We evaluate six LLMs (GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, Llama-4-Maverick) under five prompt types (zero-shot, few-shot, chain-of-thought (CoT), CoT-few-shot, self-reflection) across relevance classification and six Level-2 tasks, using accuracy, precision, recall, and F1. Results show pronounced model-prompt interaction effects: CoT-few-shot yields the most reliable precision-recall balance; zero-shot maximizes recall for high-sensitivity passes; and self-reflection underperforms due to over-inclusivity and instability across models. GPT-4o and DeepSeek provide robust overall performance, while GPT-4o-mini performs competitively at a substantially lower dollar cost. A cost-performance analysis for relevance classification (per 1,000 abstracts) reveals large absolute differences among model-prompt pairings; GPT-4o-mini remains low-cost across prompts, and structured prompts (CoT/CoT-few-shot) on GPT-4o-mini offer attractive F1 at a small incremental cost. We recommend a staged workflow that (1) deploys low-cost models with structured prompts for first-pass screening and (2) escalates only borderline cases to higher-capacity models. These findings highlight LLMs' uneven but promising potential to automate literature screening. By systematically analyzing prompt-model interactions, we provide a comparative benchmark and practical guidance for task-adaptive LLM deployment.

Figures (13)

• Figure 1: Workflow of Research Methodology
• Figure 2: Stages of SLR Process
• Figure 3: Screening Process Flowchart
• Figure 4: Distribution of Studies Relevant to SLR Automation Tasks
• Figure 5: Performance for Prompt-Model Combinations in Relevance Classification