Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SilverSight: A Multi-Task Chinese Financial Large Language Model Based on Adaptive Semantic Space Learning

Yuhang Zhou, Zeping Li, Siyu Tian, Yuchen Ni, Sen Liu, Guangnan Ye, Hongfeng Chai

TL;DR

The paper tackles multitask learning in finance with heterogeneous data, where task conflicts and data scarcity hinder performance. It introduces Adaptive Semantic Space Learning (ASSL), which uses semantic-space clustering to adaptively select LoRA experts and to perform two-stage data redistribution (A-DBSCAN density-based clustering followed by a data-sampling objective) to smooth long-tail distributions. Trained as SilverSight on 220k Chinese financial fine-tuning data, the model achieves near full-data performance with only about 10% of the data on CFLEB and FinEval benchmarks, and its adaptive expert selection closely matches the best single LoRA expert per task. The approach demonstrates efficient data usage and robust multitask performance in a specialized domain, with implications for scalable, domain-specific large language models and multi-expert systems.

Abstract

Large language models (LLMs) are increasingly being applied across various specialized fields, leveraging their extensive knowledge to empower a multitude of scenarios within these domains. However, each field encompasses a variety of specific tasks that require learning, and the diverse, heterogeneous data across these domains can lead to conflicts during model task transfer. In response to this challenge, our study introduces an Adaptive Semantic Space Learning (ASSL) framework, which utilizes the adaptive reorganization of data distributions within the semantic space to enhance the performance and selection efficacy of multi-expert models. Utilizing this framework, we trained a financial multi-task LLM named "SilverSight". Our research findings demonstrate that our framework can achieve results close to those obtained with full data training using only 10% of the data, while also exhibiting strong generalization capabilities.

SilverSight: A Multi-Task Chinese Financial Large Language Model Based on Adaptive Semantic Space Learning

TL;DR

The paper tackles multitask learning in finance with heterogeneous data, where task conflicts and data scarcity hinder performance. It introduces Adaptive Semantic Space Learning (ASSL), which uses semantic-space clustering to adaptively select LoRA experts and to perform two-stage data redistribution (A-DBSCAN density-based clustering followed by a data-sampling objective) to smooth long-tail distributions. Trained as SilverSight on 220k Chinese financial fine-tuning data, the model achieves near full-data performance with only about 10% of the data on CFLEB and FinEval benchmarks, and its adaptive expert selection closely matches the best single LoRA expert per task. The approach demonstrates efficient data usage and robust multitask performance in a specialized domain, with implications for scalable, domain-specific large language models and multi-expert systems.

Abstract

Large language models (LLMs) are increasingly being applied across various specialized fields, leveraging their extensive knowledge to empower a multitude of scenarios within these domains. However, each field encompasses a variety of specific tasks that require learning, and the diverse, heterogeneous data across these domains can lead to conflicts during model task transfer. In response to this challenge, our study introduces an Adaptive Semantic Space Learning (ASSL) framework, which utilizes the adaptive reorganization of data distributions within the semantic space to enhance the performance and selection efficacy of multi-expert models. Utilizing this framework, we trained a financial multi-task LLM named "SilverSight". Our research findings demonstrate that our framework can achieve results close to those obtained with full data training using only 10% of the data, while also exhibiting strong generalization capabilities.
Paper Structure (20 sections, 13 equations, 17 figures, 5 tables, 1 algorithm)

This paper contains 20 sections, 13 equations, 17 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. LoRA
  4. Task Definition
  5. Adaptive Semantic Space Learning
  6. Adaptive Selection of LoRA Experts
  7. Adaptive Selection of Heterogeneous Multi-source Data
  8. Experiments
  9. Data Introduction
  10. Experimental Setup
  11. SilverSight: Financial Multitask Large Model
  12. Main Results
  13. Ablation Studies
  14. Discussion
  15. Conclusion
  16. ...and 5 more sections

Figures (17)

  • Figure 1: Schematic Diagram of the SilverSight Framework: when the user inputs a financial question, select the most suitable expert to answer the question.
  • Figure 2: Adaptive Semantic Space Learning Framework.
  • Figure 3: Adaptive Semantic Space Data Redistribution Process.
  • Figure 4: Expert Selection for Each Task Category on the CFLEB Dataset
  • Figure 5: Evaluation Results of Clustering vs. Predefined Model Systems on the CFLEB Dataset
  • ...and 12 more figures