Team IELAB at TREC Clinical Trial Track 2023: Enhancing Clinical Trial Retrieval with Neural Rankers and Large Language Models
Shengyao Zhuang, Bevan Koopman, Guido Zuccon
TL;DR
The study addresses clinical trial retrieval given patient descriptions under limited labeled data by building a multi-stage pipeline that fuses neural rankers (SPLADEv2 and PubMedBERT-based dense retrievers) with a PubMedBERT-large cross-encoder and LLM-assisted components. Synthetic data generated by GPT-3.5-turbo, augmented with limited human judgments, trains robust dense and sparse retrievers, while a GPT-4-based annotator provides relevance judgments to further re-rank top results. A GPT-4 format converter mitigates distribution shift by transforming XML-style topic descriptions into natural language prompts, and the system integrates all components with score interpolation to maximize performance. The results on TREC CT 2022 and 2023 demonstrate competitive ranking improvements, with GPT-4-based judgments yielding the best NDCG@10 and P@10, illustrating a viable path to high-quality clinical trial retrieval with reduced labeling requirements and strong LLM-assisted augmentation.
Abstract
We describe team ielab from CSIRO and The University of Queensland's approach to the 2023 TREC Clinical Trials Track. Our approach was to use neural rankers but to utilise Large Language Models to overcome the issue of lack of training data for such rankers. Specifically, we employ ChatGPT to generate relevant patient descriptions for randomly selected clinical trials from the corpus. This synthetic dataset, combined with human-annotated training data from previous years, is used to train both dense and sparse retrievers based on PubmedBERT. Additionally, a cross-encoder re-ranker is integrated into the system. To further enhance the effectiveness of our approach, we prompting GPT-4 as a TREC annotator to provide judgments on our run files. These judgments are subsequently employed to re-rank the results. This architecture tightly integrates strong PubmedBERT-based rankers with the aid of SOTA Large Language Models, demonstrating a new approach to clinical trial retrieval.