Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

THaLLE-ThaiLLM: Domain-Specialized Small LLMs for Finance and Thai -- Technical Report

KBTG Labs, :, Anuruth Lertpiya, Danupat Khamnuansin, Kantapong Sucharitpongpan, Pornchanan Balee, Tawunrat Chalothorn, Thadpong Pongthawornkamol, Monchai Lertsutthiwong

TL;DR

This work tackles the challenge of delivering domain-specialized LLMs for Thai finance under privacy and resource constraints by leveraging model merging as a lightweight composition strategy. Using MergeKit, the authors independently train task-specific models (including Thai language-focused and finance-oriented variants) and merge them into multi-capability models, enabling on-premise deployment without full retraining. Two key experiments demonstrate that linear merging of Qwen3-8B with ThaiLLM-8B and THaLLE-Finance-8B yields improvements on Thai-language benchmarks (O-NET) and financial benchmarks (CFA, Thai IC), while maintaining safety and Thai-output consistency. The results support model merging as a viable, scalable approach for rapid, low-resource adaptation of open-source LLMs to domain-specific and Thai-language applications with practical impact for the financial sector.

Abstract

Large Language Models (LLMs) have demonstrated significant potential across various domains, particularly in banking and finance, where they can automate complex tasks and enhance decision-making at scale. Due to privacy, security, and regulatory concerns, organizations often prefer on-premise deployment of LLMs. The ThaiLLM initiative aims to enhance Thai language capabilities in open-LLMs, enabling Thai industry to leverage advanced language models. However, organizations often face a trade-off between deploying multiple specialized models versus the prohibitive expense of training a single multi-capability model. To address this, we explore model merging as a resource-efficient alternative for developing high-performance, multi-capability LLMs. We present results from two key experiments: first, merging Qwen-8B with ThaiLLM-8B demonstrates how ThaiLLM-8B enhances Thai general capabilities, showing an uplift of M3 and M6 O-NET exams over the general instruction-following Qwen-8B. Second, we merge Qwen-8B with both ThaiLLM-8B and THaLLE-CFA-8B. This combination results in further improvements in performance across both general and financial domains, by demonstrating an uplift in both M3 and M6 O-NET, Flare-CFA, and Thai-IC benchmarks. The report showcases the viability of model merging for efficiently creating multi-capability LLMs.

THaLLE-ThaiLLM: Domain-Specialized Small LLMs for Finance and Thai -- Technical Report

TL;DR

This work tackles the challenge of delivering domain-specialized LLMs for Thai finance under privacy and resource constraints by leveraging model merging as a lightweight composition strategy. Using MergeKit, the authors independently train task-specific models (including Thai language-focused and finance-oriented variants) and merge them into multi-capability models, enabling on-premise deployment without full retraining. Two key experiments demonstrate that linear merging of Qwen3-8B with ThaiLLM-8B and THaLLE-Finance-8B yields improvements on Thai-language benchmarks (O-NET) and financial benchmarks (CFA, Thai IC), while maintaining safety and Thai-output consistency. The results support model merging as a viable, scalable approach for rapid, low-resource adaptation of open-source LLMs to domain-specific and Thai-language applications with practical impact for the financial sector.

Abstract

Large Language Models (LLMs) have demonstrated significant potential across various domains, particularly in banking and finance, where they can automate complex tasks and enhance decision-making at scale. Due to privacy, security, and regulatory concerns, organizations often prefer on-premise deployment of LLMs. The ThaiLLM initiative aims to enhance Thai language capabilities in open-LLMs, enabling Thai industry to leverage advanced language models. However, organizations often face a trade-off between deploying multiple specialized models versus the prohibitive expense of training a single multi-capability model. To address this, we explore model merging as a resource-efficient alternative for developing high-performance, multi-capability LLMs. We present results from two key experiments: first, merging Qwen-8B with ThaiLLM-8B demonstrates how ThaiLLM-8B enhances Thai general capabilities, showing an uplift of M3 and M6 O-NET exams over the general instruction-following Qwen-8B. Second, we merge Qwen-8B with both ThaiLLM-8B and THaLLE-CFA-8B. This combination results in further improvements in performance across both general and financial domains, by demonstrating an uplift in both M3 and M6 O-NET, Flare-CFA, and Thai-IC benchmarks. The report showcases the viability of model merging for efficiently creating multi-capability LLMs.
Paper Structure (22 sections, 1 equation, 2 figures, 7 tables)

This paper contains 22 sections, 1 equation, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Ordinary National Educational Test
  4. Flare CFA
  5. Investment Consultant Exam
  6. Low-Rank Adaptation of Large Language Models
  7. Model Weight Merging
  8. Experiment
  9. Evaluation
  10. Ordinary National Educational Test
  11. Flare CFA
  12. Thai Investment Consultant
  13. ThaiSafetyBench
  14. Thai-Output Consistency Test
  15. Model Merging
  16. ...and 7 more sections

Figures (2)

  • Figure 1: ThaiLLM-8B-Instruct config
  • Figure 2: THaLLE-0.2-ThaiLLM-8B-fa config