Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoBool: An Reinforcement-Learning trained LLM for Effective Automated Boolean Query Generation for Systematic Reviews

Shuai Wang, Harrisen Scells, Bevan Koopman, Guido Zuccon

TL;DR

AutoBool introduces a reinforcement-learning framework to train LLMs for automated Boolean query generation in systematic reviews, directly optimizing retrieval performance without ground-truth target queries. It combines a retrieval-grounded reward with formatting and validity checks, enabling end-to-end optimization that balances recall and precision. The authors build a large PubMed-based dataset (PubTemp) plus a PubMed-derived training/evaluation set totaling over 65k topics, and demonstrate that AutoBool outperforms zero-shot prompts and rivals or exceeds larger GPT-based models while retrieving far fewer documents, with results that generalize to external benchmarks like CLEF TAR and Seed Collection. Ablation studies reveal key influences from backbone size, temperature, and prompt design, establishing practical guidance for deployment and future scaling in retrieval-aware query generation for evidence-based medicine.

Abstract

We present AutoBool, a reinforcement learning (RL) framework that trains large language models (LLMs) to generate effective Boolean queries for medical systematic reviews. Boolean queries are the primary mechanism for literature retrieval in this domain and must achieve high recall while maintaining reasonable precision - a challenging balance that existing prompt-based LLM approaches often struggle to achieve. A major limitation in this space is the lack of high-quality ground-truth Boolean queries for each topic, which makes supervised fine-tuning impractical. AutoBool addresses this challenge by using RL to directly optimize query generation with retrieval measures, without requiring target queries. To support this effort, we create and release the largest dataset of its kind: 65588 topics in total for training and evaluating the task of automatic Boolean query formulation. Experiments on our new dataset and two established datasets (CLEF TAR and Seed Collection) show that AutoBool significantly outperforms zero shot/few shot prompting and matches or exceeds the effectiveness of much larger GPT-based models (e.g., GPT-4o, O3) using smaller backbones. It also approaches effectiveness of expert-authored queries while retrieving 10 to 16 times fewer documents. Ablation studies reveal the critical roles of model backbone, size, decoding temperature, and prompt design. Code and data are available at https://github.com/ielab/AutoBool.

AutoBool: An Reinforcement-Learning trained LLM for Effective Automated Boolean Query Generation for Systematic Reviews

TL;DR

AutoBool introduces a reinforcement-learning framework to train LLMs for automated Boolean query generation in systematic reviews, directly optimizing retrieval performance without ground-truth target queries. It combines a retrieval-grounded reward with formatting and validity checks, enabling end-to-end optimization that balances recall and precision. The authors build a large PubMed-based dataset (PubTemp) plus a PubMed-derived training/evaluation set totaling over 65k topics, and demonstrate that AutoBool outperforms zero-shot prompts and rivals or exceeds larger GPT-based models while retrieving far fewer documents, with results that generalize to external benchmarks like CLEF TAR and Seed Collection. Ablation studies reveal key influences from backbone size, temperature, and prompt design, establishing practical guidance for deployment and future scaling in retrieval-aware query generation for evidence-based medicine.

Abstract

We present AutoBool, a reinforcement learning (RL) framework that trains large language models (LLMs) to generate effective Boolean queries for medical systematic reviews. Boolean queries are the primary mechanism for literature retrieval in this domain and must achieve high recall while maintaining reasonable precision - a challenging balance that existing prompt-based LLM approaches often struggle to achieve. A major limitation in this space is the lack of high-quality ground-truth Boolean queries for each topic, which makes supervised fine-tuning impractical. AutoBool addresses this challenge by using RL to directly optimize query generation with retrieval measures, without requiring target queries. To support this effort, we create and release the largest dataset of its kind: 65588 topics in total for training and evaluating the task of automatic Boolean query formulation. Experiments on our new dataset and two established datasets (CLEF TAR and Seed Collection) show that AutoBool significantly outperforms zero shot/few shot prompting and matches or exceeds the effectiveness of much larger GPT-based models (e.g., GPT-4o, O3) using smaller backbones. It also approaches effectiveness of expert-authored queries while retrieving 10 to 16 times fewer documents. Ablation studies reveal the critical roles of model backbone, size, decoding temperature, and prompt design. Code and data are available at https://github.com/ielab/AutoBool.
Paper Structure (61 sections, 6 equations, 11 figures, 13 tables)

This paper contains 61 sections, 6 equations, 11 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Creation
  4. Data Source and Extraction
  5. Benchmark Integrity
  6. Temporal Train-Test Split
  7. Method
  8. Reward Design
  9. Formatting Reward.
  10. Validity Reward.
  11. Retrieval Reward.
  12. Total Reward.
  13. Training Procedure
  14. Prompting Strategies
  15. Experimental Setup
  16. ...and 46 more sections

Figures (11)

  • Figure 1: Overview architecture of dataset creation and AutoBool training.
  • Figure 2: Effect of model size on the effectiveness of Boolean query generation across prompt types on the PubMed Temporal-Cutoff set, all result based on Qwen3 based Models.
  • Figure 3: Effect of reinforcement learning training temperature value on the effectiveness of Boolean query generation across prompt types on the PubMed Temporal-Cutoff set, all result based on Qwen3-4B Model.
  • Figure 4: No-reasoning prompt
  • Figure 5: Free-text Reasoning prompt with <think> and <answer> outputs
  • ...and 6 more figures