Knowledge-Driven Agentic Scientific Corpus Distillation Framework for Biomedical Large Language Models Training

TL;DR

<3-5 sentence high-level summary>The paper tackles the scarcity of high-quality biomedical training data for large language models by introducing m-KAILIN, a knowledge-driven, multi-agent framework guided by the MeSH hierarchy to autonomously distill domain-specific QA data from vast biomedical literature. It couples question generation, context retrieval, and MeSH-aware evaluation with direct preference optimization to produce AI-ready, high-quality training corpora for continuous pre-training and supervised fine-tuning. Experimental results show that models trained on the distilled data achieve state-of-the-art or competitive biomedical QA performance, including strong gains for smaller models and the ability for some open models to rival larger proprietary systems. Ablation studies confirm the complementary contributions of diverse QG agents, domain-aware retrieval, and MeSH-guided evaluation to dataset quality and downstream QA accuracy.

Abstract

Corpus distillation for biomedical large language models (LLMs) seeks to address the pressing challenge of insufficient quantity and quality in open-source annotated scientific corpora, which remains a bottleneck for effective LLM training in biomedical research. This paper proposes a knowledge-driven, agentic framework for scientific corpus distillation, tailored explicitly for LLM training in the biomedical domain, addressing the challenge posed by the complex hierarchy of biomedical knowledge. Central to our approach is a collaborative multi-agent architecture, where specialized agents, each guided by the Medical Subject Headings (MeSH) hierarchy, work in concert to autonomously extract, synthesize, and self-evaluate high-quality textual data from vast scientific literature. This agentic framework collectively generates and refines domain-specific question-answer pairs, ensuring comprehensive coverage and consistency with biomedical ontologies while minimizing manual involvement. Extensive experimental results show that language models trained on our multi-agent distilled datasets achieve notable improvements in biomedical question-answering tasks, outperforming both strong life sciences LLM baselines and advanced proprietary models. Notably, our AI-Ready dataset enables Llama3-70B to surpass GPT-4 with MedPrompt and Med-PaLM-2, despite their larger scale. Detailed ablation studies and case analyses further validate the effectiveness and synergy of each agent within the framework, highlighting the potential of multi-agent collaboration in biomedical LLM training.

Figures (14)

• Figure 1: Analyzing the limitations and challenges of the existing pipeline. The motivation of this study is to utilize the high-quality but limited annotated corpus to generate large-scale training corpora from raw scientific documents.
• Figure 2: The two distinct Question Generation Agents will generate different question by given the raw documents.
• Figure 3: The generated question pairs will be evaluated based on how well their retrieved contexts align with the knowledge hierarchy of the raw document.
• Figure 4: The fine-tuned Question Generation Agent will collaorated with Answer Generation Agent and Context Retrieval Agent to build the training corpora dataset.
• Figure 5: The prompt for continuous pre-training.