CatMemo at the FinLLM Challenge Task: Fine-Tuning Large Language Models using Data Fusion in Financial Applications
Yupeng Cao, Zhiyuan Yao, Zhi Chen, Zhiyang Deng
TL;DR
This work investigates fine-tuning large language models for financial applications via cross-task data fusion, applying PEFT with LoRA to two base models (Mistral-7B and Llama3-8B) and excluding Task3 data from fusion. The fused Task1&Task2 dataset improves performance on financial classification and summarization, with Mistral-7B delivering superior outputs to Llama3-8B, while Task3 trading performance remains poor, indicating that more advanced or larger models may be required for complex decision-making tasks. Key findings show substantial gains in Task1 and Task2 metrics when using fused data, but no consistent profitability in trading, underscoring the need for larger-scale models or alternative fusion strategies for financial trading tasks. The work highlights practical implications for deploying LLMs in finance, suggesting cautious optimism for fashioning integrated pipelines that combine learning from related textual tasks to bolster decision support.
Abstract
The integration of Large Language Models (LLMs) into financial analysis has garnered significant attention in the NLP community. This paper presents our solution to IJCAI-2024 FinLLM challenge, investigating the capabilities of LLMs within three critical areas of financial tasks: financial classification, financial text summarization, and single stock trading. We adopted Llama3-8B and Mistral-7B as base models, fine-tuning them through Parameter Efficient Fine-Tuning (PEFT) and Low-Rank Adaptation (LoRA) approaches. To enhance model performance, we combine datasets from task 1 and task 2 for data fusion. Our approach aims to tackle these diverse tasks in a comprehensive and integrated manner, showcasing LLMs' capacity to address diverse and complex financial tasks with improved accuracy and decision-making capabilities.