Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance

TL;DR

This study tackles domain specialization for LLMs by investigating continual pre-training (CPT) model merging to build finance-focused, multi-skill LLMs. It introduces a principled three-stage framework to analyze knowledge recovery, cross-domain complementarity, and emergent capabilities, using a base model $ heta_0$ and CPT experts in Finance, Math, and Japanese with merge operators $f_a$ under hyperparameters $b gamma$. The authors compare Task Arithmetic (TA), TIES (TI), and DARE-TIES (DA) on a financial benchmark of 18 tasks from 8 datasets, finding that base+CPT merges recover lost general knowledge, dual CPT merges yield emergent cross-domain skills (notably Finance+Math), while tri-CPT mergers often degrade performance due to interference. TA delivers the strongest gains but is highly sensitive to hyperparameters, whereas TI offers more robust and stable improvements; similarity between constituent models correlates with gains but does not reliably predict emergent capabilities. Overall, CPT model merging provides a resource-efficient path to multi-skilled LLMs and offers principled guidance for constructing domain-specialized models from existing assets, with implications for cross-domain reasoning and multilingual transfer.

Abstract

While LLMs excel at general tasks, they struggle in specialized domains like finance, requiring diverse skills in domain knowledge, mathematical reasoning, and multilingual processing. Merging domain-specific Continual Pre-training (CPT) "experts" offers a practical alternative to costly and unstable multi-skill training. However, unlike established Supervised Fine-Tuning (SFT) model-based merging, CPT model merging remains largely unexplored. We address this gap by creating financial LLMs from experts in finance, math, and Japanese. We propose a three-stage evaluation focusing on knowledge recovery, complementarity, and emergence, and assess three merging methods (Task Arithmetic, TIES, and DARE-TIES) on a comprehensive financial benchmark curated from 18 tasks across 8 established datasets. Results show that merging an expert with its base model recovers general knowledge lost during CPT, while merging experts improves performance and can yield emergent cross-domain skills. Among the methods, Task Arithmetic performs strongly but is hyperparameter-sensitive, whereas TIES is more robust. Our findings also suggest that while model similarity correlates with merging success, emergent skills depend on more complex factors. This work presents the first foundational analysis of CPT model merging, establishing a principled framework and providing clear guidance for building multi-skill LLMs from existing assets.

Figures (7)

• Figure 1: Three-stage model merging framework designed to analyze knowledge transfer, complementarity, and interference in CPT model integration.
• Figure 2: Task-level effects of CPT-based model merging. The left heatmap shows Gain (improvement over the constituent average), and the right shows Outperform Gap (OG; improvement over the best constituent).
• Figure 3: Model-wise answer sheets for the CS-CoT task. (Blue = correct, White = incorrect).
• Figure 4: Overall performance across different merge strategies (TA, TI, DA) under varying hyperparameter values. TA sweeps the scaling coefficient $\lambda$, while TI and DA sweep the sparsity/density parameter $d$.
• Figure 5: Relationship between the L2 distance and cosine similarity of pre-merge models and the resulting performance (Macro-Gain and Macro-OG). Macro-Gain is higher when the two models are more similar (smaller L2, larger cosine; Spearman’s rank correlation, two-sided $p < 0.05$), but we do not see a clear pattern for Macro-OG ($p > 0.60$).