AiReview: An Open Platform for Accelerating Systematic Reviews with LLMs

TL;DR

The paper tackles the time-intensive nature of title and abstract screening in systematic reviews by enabling end-user access to LLMs through AiReview, an extensible framework and web interface. It details a Docker-based, API-integrated architecture that supports configurable LLM roles (pre-reviewer, co-reviewer, post-reviewer) and editable prompts to guide screening, along with a SR Assistant that aids decision-making and quality control. A framework for categorizing LLM use cases by role and interaction level is proposed, including an effort-savings ranking across seven pipelines and real-world scenarios such as teaching and resource-limited teams. The work aims to bridge cutting-edge LLM-assisted screening with practical SR production, emphasizing transparency, scalability, and bias considerations, with planned user studies and expansion to additional SR tasks and collaborative workflows.

Abstract

Systematic reviews are fundamental to evidence-based medicine. Creating one is time-consuming and labour-intensive, mainly due to the need to screen, or assess, many studies for inclusion in the review. Several tools have been developed to streamline this process, mostly relying on traditional machine learning methods. Large language models (LLMs) have shown potential in further accelerating the screening process. However, no tool currently allows end users to directly leverage LLMs for screening or facilitates systematic and transparent usage of LLM-assisted screening methods. This paper introduces (i) an extensible framework for applying LLMs to systematic review tasks, particularly title and abstract screening, and (ii) a web-based interface for LLM-assisted screening. Together, these elements form AiReview-a novel platform for LLM-assisted systematic review creation. AiReview is the first of its kind to bridge the gap between cutting-edge LLM-assisted screening methods and those that create medical systematic reviews. The tool is available at https://aireview.ielab.io. The source code is also open sourced at https://github.com/ielab/ai-review.

Figures (2)

• Figure 1: Holistic architecture and workflow of AiReview.
• Figure 2: In the screening interface (a), users can select studies to expand for abstract screening, indicated with a purple edge. They have the option to include or exclude the selected study for further detailed review. LLM suggestions are immediately visible, as the settings are configured for Pre-reviewer and high LLM interaction. When Co-reviewer is enabled, users can engage the SR assistant by clicking the 'Ask AI' button , which reveals interactive features within the right panel . This panel includes three tabs: 'Chat', 'Model Config' (b) and 'Prompts' (c) for interacting with the LLM, adjusting model settings, and editing prompts, respectively. Users can start a new chat via . The response area has LLM feedback for inclusion based on the interaction level. In 'low' mode, users are limited to interact with the LLM by predefined options, i.e., PICO Extraction and Detailed Reasoning , while in 'high' mode , users can directly prompt the LLM using . The 'Model Config' tab allows users to change the LLM model, temperature, and response settings. The 'Prompts' tab enables users to edit LLM prompts about the objective and persona, instructions in a task template, response format, and inclusion criteria.