The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Merging

TL;DR

The paper tackles the challenge of building finance-focused instruction-tuned LLMs without instruction data by first continually pretraining a general-purpose Japanese LLM on a finance-specific corpus and then merging it with a general-purpose instruction-tuned LLM using a simple weight-interpolation scheme. It demonstrates that instruction-support and domain knowledge act largely independently in task arithmetic, enabling effective construction of domain-specific instruction-tuned models without new instruction data. Empirical results on Japanese financial benchmarks and pfmt-bench-fin-ja show that the domain-specific instruction-tuned model achieves the best overall performance, with the approach offering a cost-efficient pathway to domain specialization. Translation performance lags in some tasks, highlighting the need for multilingual data, but the method provides a practical route for deploying finance-focused LLMs using publicly available models.

Abstract

This paper proposes a novel method for constructing instruction-tuned large language models (LLMs) for finance without instruction data. Traditionally, developing such domain-specific LLMs has been resource-intensive, requiring a large dataset and significant computational power for continual pretraining and instruction tuning. Our study proposes a simpler approach that combines domain-specific continual pretraining with model merging. Given that general-purpose pretrained LLMs and their instruction-tuned LLMs are often publicly available, they can be leveraged to obtain the necessary instruction task vector. By merging this with a domain-specific pretrained vector, we can effectively create instruction-tuned LLMs for finance without additional instruction data. Our process involves two steps: first, we perform continual pretraining on financial data; second, we merge the instruction-tuned vector with the domain-specific pretrained vector. Our experiments demonstrate the successful construction of instruction-tuned LLMs for finance. One major advantage of our method is that the instruction-tuned and domain-specific pretrained vectors are nearly independent. This independence makes our approach highly effective. The Japanese financial instruction-tuned LLMs we developed in this study are available at https://huggingface.co/pfnet/nekomata-14b-pfn-qfin-inst-merge.

Figures (4)

• Figure 1: Overview of Our Method
• Figure 2: Loss Curve
• Figure 3: Evaluation Results (Radar Chart) of pfmt-bench-fin-ja
• Figure 4: Weight Similarities between the Domain-specific and Instruction Task Arithmetic Vectors of the Merged Model