TheBlueScrubs-v1, a comprehensive curated medical dataset derived from the internet

TL;DR

TheBlueScrubs-v1 addresses the shortage of large-scale, diverse clinical text by assembling roughly $25$ billion tokens from SlimPajama through a scalable two-stage filtering pipeline. A TF-IDF-based linear classifier filters medical content (AUC ≈ $0.95$) before a large-language-model-based evaluation with Llama 3.1 (70B) provides three quality scores—medical relevance, factual precision, and safety—validated against clinician judgments and GPT-4o; a separate cancer-classification pipeline marks about $11$ billion oncology tokens. Two demonstrations show practical utility: a ModernBERT-based safety classifier achieving AUC $0.9642$ and a fine-tuned $8$B Llama model on a high-quality subset that matches or surpasses UMLS-based baselines on medical tasks. The dataset, openly available, enables scalable pretraining, safety testing, and subdomain research for clinical LLMs, with plans to extend to larger corpora and multilingual content, thereby complementing traditional sources like PubMed and UMLS with broad internet-derived medical discourse.

Abstract

The need for robust and diverse data sets to train clinical large language models (cLLMs) is critical given that currently available public repositories often prove too limited in size or scope for comprehensive medical use. While resources like PubMed provide foundational medical literature, they capture only a narrow range of formal publications and omit the broader medical discourse on the internet. To address these deficits, we introduce TheBlueScrubs-v1, a curated dataset of over 25 billion medical tokens - nearly three times larger than PubMed - drawn from a broad-scale internet corpus. Our two-stage filtering pipeline employs a Logistic Regression model for document screening (achieving an AUC of approximately 0.95 on external validation), followed by verification via a 70B-parameter Llama 3.1 instruct model. Each text is assigned three LLM-based quality scores encompassing medical relevance, precision and factual detail, and safety and ethical standards. Clinician reviews confirm high concordance with these automated evaluations, and a specialized cancer classifier further labels approximately 11 billion oncology tokens. Two demonstration tasks highlight the dataset's practical value: first, we distill the safety evaluations to a smaller BERT-style model that reaches an AUC near 0.96 on unseen data; second, we fine-tune a compact LLM on a filtered subset, showing measurable improvements over standard baselines in medical benchmarks as well as private ones. This Data Descriptor details the dataset's creation and validation, underscoring its potential utility for medical AI research.

Figures (8)

• Figure 1: The Blue Scrubs creation process through filtering, evaluation, and labeling.
• Figure 2: Prompt template used during Llama 3.1 (70B) inference.
• Figure 3: Histogram for each evaluation category (Scope, Precision, Safety).
• Figure 4: Mean and standard deviation of Scope of Medical Relevance scores from the clinician survey (X) vs. Llama 3.1 (70B) (Y).
• Figure 5: Mean and standard deviation of Scope of Medical Relevance scores from GPT 4o (X) vs. Llama 3.1 (70B) (Y).