DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining

TL;DR

The paper addresses the scarcity of high-quality, domain-specific pre-training data for adapting LLMs to specialized industries. It introduces DoPAMine, a seed-guided data mining framework that uses an LLM to generate diverse domain seeds and retrieves real-world domain data from Common Crawl via embedding-based nearest-neighbor search, followed by multi-label domain classification and continued pre-training. Experiments with 7B-LM CPT on healthcare and finance demonstrate consistent improvements across multiple benchmarks in zero-shot and few-shot settings, with quantified gains such as 4.9% (zero-shot) and 5.1% (5-shot) for healthcare, and 2.9% (zero-shot) and 6.7% (5-shot) for finance. The approach offers a scalable, controllable alternative to purely synthetic data, mitigating factuality concerns while enhancing domain specialization and enabling practical adoption in data-scarce sectors.

Abstract

Large Language Models (LLMs) have shown remarkable ability to generalize effectively across numerous industry domains while executing a range of tasks. Many of these competencies are obtained from the data utilized during the pre-training phase of the Language Models (LMs). However, these models exhibit limitations when tasked with performing in specialized or low-resource industry domains. More recent approaches use LLMs for generating domain-specific synthetic data but most often they lack in truthfulness and complexity. Alternatively, in cases where domain data is available like healthcare and finance most of the LMs are proprietary necessitating the need for a scalable method to curate real world industry specific pre-training data. In this work, we propose an automated and scalable framework - DoPAMine:Domain-specific Pre-training Adaptation from seed-guided data Mining, to mine domain specific training data from a large data corpus for domain adaptation of a LM. The framework leverages the parametric knowledge of a LLM to generate diverse and representative seed data tailored to a specific domain which is then used to mine real world data from a large data corpus like Common Crawl. We evaluated our framework's performance in the continual pre-training (CPT) setting by training two domain specific 7B parameter LMs in healthcare and finance with data mined via DoPAMine. Our experiments show that DoPAMine boosts the performance of pre-trained LLMs on average by 4.9% and 5.1% in zero-shot and 5-shot settings respectively on healthcare tasks from MMLU, MedQA, MedMCQA and PubMedQA datasets, and 2.9% and 6.7% for zero-shot and 5-shot settings respectively on finance tasks from FiQA-SA, FPB and Headlines datasets when compared to the baseline.

Figures (5)

• Figure 1: DoPAMine components for automated and scalable mining of domain specific data
• Figure 2: Seed data generation prompt
• Figure 3: UMAP of mined in-domain data
• Figure 4: Results comparison
• Figure 5: LLM as a judge: Prompt