Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluation

Shamane Siriwardhana, Mark McQuade, Thomas Gauthier, Lucas Atkins, Fernando Fernandes Neto, Luke Meyers, Anneketh Vij, Tyler Odenthal, Charles Goddard, Mary MacCarthy, Jacob Solawetz

TL;DR

The paper investigates domain adaptation of Meta-Llama-3-70B-Instruct to SEC data using continual pre-training (CPT) and model merging to boost domain-specific capabilities while preserving general instruction abilities. It reports an intermediate 20B-token CPT checkpoint derived from a 70B-parameter model trained on SEC data and a 1B-token general data mix, with evaluation across domain-specific and general benchmarks. The study demonstrates reduced domain perplexity through CPT and shows that merging with the base instruct model via TIES/MergeKit helps recover general capabilities, addressing catastrophic forgetting. The findings validate the feasibility of finance/regulatory domain-specific chat agents and outline future directions, including alignment with SFT, DPO, RLHF, improved data processing, and more advanced merging strategies.

Abstract

We conducted extensive experiments on domain adaptation of the Meta-Llama-3-70B-Instruct model on SEC data, exploring its performance on both general and domain-specific benchmarks. Our focus included continual pre-training (CPT) and model merging, aiming to enhance the model's domain-specific capabilities while mitigating catastrophic forgetting. Through this study, we evaluated the impact of integrating financial regulatory data into a robust language model and examined the effectiveness of our model merging techniques in preserving and improving the model's instructive abilities. The model is accessible at hugging face: https://huggingface.co/arcee-ai/Llama-3-SEC-Base, arcee-ai/Llama-3-SEC-Base. This is an intermediate checkpoint of our final model, which has seen 20B tokens so far. The full model is still in the process of training. This is a preprint technical report with thorough evaluations to understand the entire process.

Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluation

TL;DR

The paper investigates domain adaptation of Meta-Llama-3-70B-Instruct to SEC data using continual pre-training (CPT) and model merging to boost domain-specific capabilities while preserving general instruction abilities. It reports an intermediate 20B-token CPT checkpoint derived from a 70B-parameter model trained on SEC data and a 1B-token general data mix, with evaluation across domain-specific and general benchmarks. The study demonstrates reduced domain perplexity through CPT and shows that merging with the base instruct model via TIES/MergeKit helps recover general capabilities, addressing catastrophic forgetting. The findings validate the feasibility of finance/regulatory domain-specific chat agents and outline future directions, including alignment with SFT, DPO, RLHF, improved data processing, and more advanced merging strategies.

Abstract

We conducted extensive experiments on domain adaptation of the Meta-Llama-3-70B-Instruct model on SEC data, exploring its performance on both general and domain-specific benchmarks. Our focus included continual pre-training (CPT) and model merging, aiming to enhance the model's domain-specific capabilities while mitigating catastrophic forgetting. Through this study, we evaluated the impact of integrating financial regulatory data into a robust language model and examined the effectiveness of our model merging techniques in preserving and improving the model's instructive abilities. The model is accessible at hugging face: https://huggingface.co/arcee-ai/Llama-3-SEC-Base, arcee-ai/Llama-3-SEC-Base. This is an intermediate checkpoint of our final model, which has seen 20B tokens so far. The full model is still in the process of training. This is a preprint technical report with thorough evaluations to understand the entire process.
Paper Structure (13 sections, 6 figures)

This paper contains 13 sections, 6 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Domain adaptation of LLMs
  4. Continual Pre-Training
  5. Catastrophic Forgetting
  6. Model Merging
  7. Data Acquisition and Pre-Processing
  8. Continual Pre-Training (CPT) with Megatron-Core
  9. Post CPT Model Merging with MergeKit
  10. Evaluation
  11. Domain-Specific Evaluations Metrics
  12. General Evaluations Metrics
  13. Conclusion and Future Work

Figures (6)

  • Figure 1: LM Loss
  • Figure 2: Learning Rate
  • Figure 3: Domain Specific Perplexity of Model Variants (lower the better)
  • Figure 4: Domain Specific Evaluations of Model Variants
  • Figure 5: General Evaluations of Model Variants
  • ...and 1 more figures