Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Hierarchy Guided Biological-Medical Dataset Distillation for Domain LLM Training

Xunxin Cai, Chengrui Wang, Qingqing Long, Yuanchun Zhou, Meng Xiao

TL;DR

This work addresses the scarcity and uneven quality of open biomedical datasets for training domain LLMs. It introduces KAILIN, an automated pipeline that distills domain-aligned data by generating questions with two QA models, retrieving rich contexts from PubMed, and selecting questions via MeSH-based knowledge-hierarchy similarity, followed by direct preference optimization and ideal dataset construction. The resulting AI-Ready and I2 datasets enable improved downstream QA performance across model sizes, with small general-purpose models approaching or surpassing domain-specific baselines and a 70B model outperforming GPT-4 in certain settings. By explicitly integrating biomedical knowledge hierarchies, the approach enhances data quality and transfer efficiency for domain-aware LLM training.

Abstract

The rapid advancement of large language models (LLMs) in biological-medical applications has highlighted a gap between their potential and the limited scale and often low quality of available open-source annotated textual datasets. In addition, the inherent complexity of the biomedical knowledge hierarchy significantly hampers efforts to bridge this gap.Can LLMs themselves play a pivotal role in overcoming this limitation? Motivated by this question, we investigate this challenge in the present study.We propose a framework that automates the distillation of high-quality textual training data from the extensive scientific literature. Our approach self-evaluates and generates questions that are more closely aligned with the biomedical domain, guided by the biomedical knowledge hierarchy through medical subject headings (MeSH). This comprehensive framework establishes an automated workflow, thereby eliminating the need for manual intervention. Furthermore, we conducted comprehensive experiments to evaluate the impact of our framework-generated data on downstream language models of varying sizes. Our approach substantially improves question-answering tasks compared to pre-trained models from the life sciences domain and powerful close-source models represented by GPT-4. Notably, the generated AI-Ready dataset enabled the Llama3-70B base model to outperform GPT-4 using MedPrompt with multiple times the number of parameters. Detailed case studies and ablation experiments underscore the significance of each component within our framework

Knowledge Hierarchy Guided Biological-Medical Dataset Distillation for Domain LLM Training

TL;DR

This work addresses the scarcity and uneven quality of open biomedical datasets for training domain LLMs. It introduces KAILIN, an automated pipeline that distills domain-aligned data by generating questions with two QA models, retrieving rich contexts from PubMed, and selecting questions via MeSH-based knowledge-hierarchy similarity, followed by direct preference optimization and ideal dataset construction. The resulting AI-Ready and I2 datasets enable improved downstream QA performance across model sizes, with small general-purpose models approaching or surpassing domain-specific baselines and a 70B model outperforming GPT-4 in certain settings. By explicitly integrating biomedical knowledge hierarchies, the approach enhances data quality and transfer efficiency for domain-aware LLM training.

Abstract

The rapid advancement of large language models (LLMs) in biological-medical applications has highlighted a gap between their potential and the limited scale and often low quality of available open-source annotated textual datasets. In addition, the inherent complexity of the biomedical knowledge hierarchy significantly hampers efforts to bridge this gap.Can LLMs themselves play a pivotal role in overcoming this limitation? Motivated by this question, we investigate this challenge in the present study.We propose a framework that automates the distillation of high-quality textual training data from the extensive scientific literature. Our approach self-evaluates and generates questions that are more closely aligned with the biomedical domain, guided by the biomedical knowledge hierarchy through medical subject headings (MeSH). This comprehensive framework establishes an automated workflow, thereby eliminating the need for manual intervention. Furthermore, we conducted comprehensive experiments to evaluate the impact of our framework-generated data on downstream language models of varying sizes. Our approach substantially improves question-answering tasks compared to pre-trained models from the life sciences domain and powerful close-source models represented by GPT-4. Notably, the generated AI-Ready dataset enabled the Llama3-70B base model to outperform GPT-4 using MedPrompt with multiple times the number of parameters. Detailed case studies and ablation experiments underscore the significance of each component within our framework
Paper Structure (16 sections, 4 equations, 2 figures, 1 table)

This paper contains 16 sections, 4 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Fine-Tuning Question Generator:
  4. Retrieval Process:
  5. Knowledge Hierarchy Similarity Evaluation:
  6. Preference Dataset Construction.
  7. Direct Preference Optimization.
  8. Ideal Dataset Construction
  9. Training for Downstream Task
  10. Experiment
  11. Experimental Setups
  12. Evaluation Datasets.
  13. Main Results and Analysis
  14. Ablation Study
  15. Conclusion
  16. ...and 1 more sections

Figures (2)

  • Figure 1: The overview of KAILIN framework.
  • Figure 2: Evaluations (accuracy (%)) for PubMedQA jin2019pubmedqa problems on our models compared to other open-source models, closed-source models, and domain-specific models. Models marked with * indicate that they are domain-specific large models focused on the biomedical field rather than general-purpose models. The # symbol denotes closed-source models as opposed to open-source ones.