Learning Beyond the Surface: How Far Can Continual Pre-Training with LoRA Enhance LLMs' Domain-Specific Insight Learning?

TL;DR

This work probes how far continual pre-training with LoRA can enable LLMs to internalize domain-specific insights across medicine and finance, focusing on declarative, statistical, and probabilistic types. By extracting structured <subject–relation–object> triples from domain datasets and evaluating with 500 queries per dataset, the authors compare training on original documents versus simplified triple content. They find that training on simplified triples yields substantial gains in insight learning, particularly for declarative and statistical insights, while probabilistic insights remain challenging; larger models further enhance learning. The findings highlight the importance of input format and data curation in enabling scalable domain-insight learning for LLMs, informing future continual pre-training strategies and dataset design.

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance on various tasks, yet their ability to extract and internalize deeper insights from domain-specific datasets remains underexplored. In this study, we investigate how continual pre-training can enhance LLMs' capacity for insight learning across three distinct forms: declarative, statistical, and probabilistic insights. Focusing on two critical domains: medicine and finance, we employ LoRA to train LLMs on two existing datasets. To evaluate each insight type, we create benchmarks to measure how well continual pre-training helps models go beyond surface-level knowledge. We also assess the impact of document modification on capturing insights. The results show that, while continual pre-training on original documents has a marginal effect, modifying documents to retain only essential information significantly enhances the insight-learning capabilities of LLMs.

Figures (4)

• Figure 1: We use domain-specific data, like the Hallmarks of Cancer dataset, to adapt an LLM through continued pre-training with LoRA. Our goal is to assess whether LLMs are capable of effectively capture three types of insights: declarative, statistical, and probabilistic.
• Figure 2: LLMs performance on insight extraction during continual pre-training. Declarative and statistical insights show slight improvement, while probabilistic insights remain largely unchanged. Increasing model size and using simplified documents significantly enhance performance.
• Figure 3: Distribution of the number of objects for statistical insights and probability values $p(\text{entity}_2|\text{entity}_1)$ for probabilistic insights in the created evaluation sets of each dataset.
• Figure 4: LLM performance on insight extraction during continual pre-training. We report F1 scores for declarative insights, Recall@5 for statistical insights, and Pearson correlation coefficients for probabilistic insights. The results follow similar trends to the metrics in Figure \ref{['fig:res']}.