A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classification

TL;DR

This study investigates the efficacy of instruction fine-tuning smaller-scale LLMs, including Mistral-7B, Llama3-8B, and Phi3-mini, to enhance their performance in financial text classification tasks, and underscores the effectiveness of instruction fine-tuning and model merging for adapting LLMs to specialized financial text classification tasks.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities across diverse Natural Language Processing (NLP) tasks, including language understanding, reasoning, and generation. However, general-domain LLMs often struggle with financial tasks due to the technical and specialized nature of financial texts. This study investigates the efficacy of instruction fine-tuning smaller-scale LLMs, including Mistral-7B, Llama3-8B, and Phi3-mini, to enhance their performance in financial text classification tasks. We fine-tuned both instruction-tuned and base models across four financial classification tasks, achieving significant improvements in task-specific performance. Furthermore, we evaluated the zero-shot capabilities of these fine-tuned models on three unseen complex financial tasks, including argument classification, deal completeness classification, and causal classification. Our results indicate while base model fine-tuning led to greater degradation, instruction-tuned models maintained more robust performance. To address this degradation, we employed model merging techniques, integrating single-task domain-specific fine-tuned models with the base model. Using this merging method resulted in significant enhancements in zero-shot performance, even exceeding the original model's accuracy on certain datasets. Our findings underscore the effectiveness of instruction fine-tuning and model merging for adapting LLMs to specialized financial text classification tasks.

Figures (9)

• Figure 1: The three approaches used in this work a) single task fine-tuning, b)Multi-task fine-tuning c)Merging with vanilla models
• Figure 2: Few-shot experiment results (Accuracy) on the FBP, FiQA-SA, Headline-Dir, and FOMC datasets using the Llama3-8B, Mistral-7B, and Phi-3-mini instruct models. Results for the FinRED dataset are excluded due to its large label space, which resulted in a low performance (below 5%).
• Figure 3: Performance comparison of vanilla models (zero-shot instruct models), multi-task fine-tuned instruct models, multi-task fine-tuned base models, and GPT-4 models on five financial classification datasets for Llama3-8B and Mistral-7B models. F1 score is reported.
• Figure 4: Performance comparison of vanilla models (zero-shot instruct models), multi-task fine-tuned instruct models, and GPT-4 models on five financial classification datasets for Phi-3 model. F1 score is reported.
• Figure 5: Performance comparison of vanilla models (zero-shot instruct models), multi-task fine-tuned instruct models, multi-task fine-tuned base models, and merged models on three unseen datasets for Llama3-8B and Mistral-7B models. F1 score is reported.