AI-assisted Protocol Information Extraction For Improved Accuracy and Efficiency in Clinical Trial Workflows

TL;DR

This work introduces a clinical-trial-specific Retrieval-Augmented Generation (RAG) pipeline to extract structured protocol information from long and heterogeneous trial documents. By combining document processing, element-focused retrieval, structured generation, and a vision-enabled SoE extraction component, the approach achieves higher accuracy than standalone LLMs and significantly reduces data extraction time. An LLM-adjudicated, hybrid annotation framework and a controlled CRC experiment establish robustness of ground truth and demonstrate practical benefits in efficiency, data quality, and user satisfaction. The findings support deploying AI-assisted protocol information extraction within real-world workflows to improve start-up, monitoring, and downstream automation while maintaining auditability and regulatory compliance.

Abstract

Increasing clinical trial protocol complexity, amendments, and challenges around knowledge management create significant burden for trial teams. Structuring protocol content into standard formats has the potential to improve efficiency, support documentation quality, and strengthen compliance. We evaluate an Artificial Intelligence (AI) system using generative LLMs with Retrieval-Augmented Generation (RAG) for automated clinical trial protocol information extraction. We compare the extraction accuracy of our clinical-trial-specific RAG process against that of publicly available (standalone) LLMs. We also assess the operational impact of AI-assistance on simulated extraction CRC workflows. Our RAG process was measured as more accurate (87.8%) than standalone LLMs with fine-tuned prompts (62.6%) against expert-supported reference annotations. In the simulated extraction workflows, AI-assisted tasks were completed 40% faster, rated as less cognitively demanding and strongly preferred by users. While expert oversight remains essential, this suggests that AI-assisted extraction can enable protocol intelligence at scale, motivating the integration of similar methodologies into real world clinical workflows to further validate its impact on feasibility, study start-up, and post-activation monitoring.

Figures (3)

• Figure 1: RAG Process for Clinical Protocol Information Extraction. The RAG system first processes protocol Portable Document Format (PDF) files subdividing them into meaningful chunks and storing them in a vector database with semantic embeddings. When users query for specific protocol information (e.g., inclusion/exclusion criteria), the system retrieves the most relevant chunks and provides them as context to a LLM along with specialized instructions. The LLM then generates a (semi-)structured, standardized output that research teams can readily use for protocol review and study operations.
• Figure 2: Comparison of task completion times between non-AI and AI-assisted protocol abstraction tasks. Box plot shows median completion time, interquartile range, and outliers (circles) for both conditions. Non-AI tasks required substantially longer completion times compared to AI-assisted tasks.
• Figure 3: Distribution and comparison of item-weighted accuracy scores between AI-assisted and unassisted conditions. (A) Histogram showing count distribution of item-weighted scores , with unassisted scores (blue) and unassisted scores (orange) overlaid. (B) Box plot displaying median, quartiles, and range of scores for both conditions. (C) Count distribution by rounded score values on 0-5 scale, comparing frequencies between AI and non-AI groups. (D) Kernel density estimation curves showing smoothed probability distributions for both conditions.