Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NumLLM: Numeric-Sensitive Large Language Model for Chinese Finance

Huan-Yi Su, Ke Wu, Yu-Hao Huang, Wu-Jun Li

TL;DR

NumLLM tackles the challenge of numeric-variable understanding in Chinese finance by leveraging a textbook-derived financial corpus and a dual-LoRA fine-tuning strategy. It trains one LoRA module for domain-adaptive continual pre-training and another for numeric-variable reasoning (NumCT), then merges them via an SVD-based fusion to form a numeric-sensitive foundation for inference. On the FinEval benchmark, NumLLM achieves the best overall accuracy, with notable gains on numeric and non-numeric questions across multiple sub-domains, and ablation studies validate the necessity of each component and the fusion method. The approach offers a practical, scalable route to enhancing numeric capability in FinLLMs and has potential for cross-language extension.

Abstract

Recently, many works have proposed various financial large language models (FinLLMs) by pre-training from scratch or fine-tuning open-sourced LLMs on financial corpora. However, existing FinLLMs exhibit unsatisfactory performance in understanding financial text when numeric variables are involved in questions. In this paper, we propose a novel LLM, called numeric-sensitive large language model (NumLLM), for Chinese finance. We first construct a financial corpus from financial textbooks which is essential for improving numeric capability of LLMs during fine-tuning. After that, we train two individual low-rank adaptation (LoRA) modules by fine-tuning on our constructed financial corpus. One module is for adapting general-purpose LLMs to financial domain, and the other module is for enhancing the ability of NumLLM to understand financial text with numeric variables. Lastly, we merge the two LoRA modules into the foundation model to obtain NumLLM for inference. Experiments on financial question-answering benchmark show that NumLLM can boost the performance of the foundation model and can achieve the best overall performance compared to all baselines, on both numeric and non-numeric questions.

NumLLM: Numeric-Sensitive Large Language Model for Chinese Finance

TL;DR

NumLLM tackles the challenge of numeric-variable understanding in Chinese finance by leveraging a textbook-derived financial corpus and a dual-LoRA fine-tuning strategy. It trains one LoRA module for domain-adaptive continual pre-training and another for numeric-variable reasoning (NumCT), then merges them via an SVD-based fusion to form a numeric-sensitive foundation for inference. On the FinEval benchmark, NumLLM achieves the best overall accuracy, with notable gains on numeric and non-numeric questions across multiple sub-domains, and ablation studies validate the necessity of each component and the fusion method. The approach offers a practical, scalable route to enhancing numeric capability in FinLLMs and has potential for cross-language extension.

Abstract

Recently, many works have proposed various financial large language models (FinLLMs) by pre-training from scratch or fine-tuning open-sourced LLMs on financial corpora. However, existing FinLLMs exhibit unsatisfactory performance in understanding financial text when numeric variables are involved in questions. In this paper, we propose a novel LLM, called numeric-sensitive large language model (NumLLM), for Chinese finance. We first construct a financial corpus from financial textbooks which is essential for improving numeric capability of LLMs during fine-tuning. After that, we train two individual low-rank adaptation (LoRA) modules by fine-tuning on our constructed financial corpus. One module is for adapting general-purpose LLMs to financial domain, and the other module is for enhancing the ability of NumLLM to understand financial text with numeric variables. Lastly, we merge the two LoRA modules into the foundation model to obtain NumLLM for inference. Experiments on financial question-answering benchmark show that NumLLM can boost the performance of the foundation model and can achieve the best overall performance compared to all baselines, on both numeric and non-numeric questions.
Paper Structure (24 sections, 10 equations, 4 figures, 4 tables)

This paper contains 24 sections, 10 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Financial Corpora
  4. Financial LLMs
  5. Numeric-Sensitive Large Language Model
  6. Fin-Textbooks: Chinese Financial Textbook Corpus
  7. Continual Pre-Training
  8. Numeric-Sensitive Choice Tuning
  9. Numeric-Sensitive Instance Extraction
  10. Numeric-Masked Choice Generation
  11. NumCT Instruction Construction
  12. Instruction Fine-Tuning
  13. Mixing and Merging LoRA Modules
  14. Experiment
  15. Experimental Setup
  16. ...and 9 more sections

Figures (4)

  • Figure 1: The architecture of NumLLM.
  • Figure 2: An example of instruction-output pair constructed by NumCT. Translation in English is provided below the original text.
  • Figure 3: Examples of numeric and non-numeric questions. Translation in English is provided below the original text.
  • Figure 4: A radar graph for average (over numeric and non-numeric questions) accuracy (%) of all financial LLMs in all sub-domains.