Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasets

Dannong Wang, Jaisal Patel, Daochen Zha, Steve Y. Yang, Xiao-Yang Liu

TL;DR

FinLoRA introduces a comprehensive benchmark to evaluate LoRA-based PEFT methods for fine-tuning LLMs on financial tasks, including four novel XBRL analysis datasets. It compares five LoRA variants across five base LLMs, using 19 public and newly-created financial datasets, and reports accuracy, F1, and BERTScore with cost metrics. The study finds that LoRA methods yield on average a 36% accuracy improvement over baselines, particularly for XBRL analysis tasks that benefit from standardized semantics, while privacy and resource considerations are examined through Federated LoRA and detailed cost analyses. The results provide practical guidance for deploying LoRA in finance and offer an open-source platform to accelerate future research in specialized financial NLP.

Abstract

Low-rank adaptation (LoRA) methods show great potential for scaling pre-trained general-purpose Large Language Models (LLMs) to hundreds or thousands of use scenarios. However, their efficacy in high-stakes domains like finance is rarely explored, e.g., passing CFA exams and analyzing SEC filings. In this paper, we present the open-source FinLoRA project that benchmarks LoRA methods on both general and highly professional financial tasks. First, we curated 19 datasets covering diverse financial applications; in particular, we created four novel XBRL analysis datasets based on 150 SEC filings. Second, we evaluated five LoRA methods and five base LLMs. Finally, we provide extensive experimental results in terms of accuracy, F1, and BERTScore and report computational cost in terms of time and GPU memory during fine-tuning and inference stages. We find that LoRA methods achieved substantial performance gains of 36\% on average over base models. Our FinLoRA project provides an affordable and scalable approach to democratize financial intelligence to the general public. Datasets, LoRA adapters, code, and documentation are available at https://github.com/Open-Finance-Lab/FinLoRA

FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasets

TL;DR

FinLoRA introduces a comprehensive benchmark to evaluate LoRA-based PEFT methods for fine-tuning LLMs on financial tasks, including four novel XBRL analysis datasets. It compares five LoRA variants across five base LLMs, using 19 public and newly-created financial datasets, and reports accuracy, F1, and BERTScore with cost metrics. The study finds that LoRA methods yield on average a 36% accuracy improvement over baselines, particularly for XBRL analysis tasks that benefit from standardized semantics, while privacy and resource considerations are examined through Federated LoRA and detailed cost analyses. The results provide practical guidance for deploying LoRA in finance and offer an open-source platform to accelerate future research in specialized financial NLP.

Abstract

Low-rank adaptation (LoRA) methods show great potential for scaling pre-trained general-purpose Large Language Models (LLMs) to hundreds or thousands of use scenarios. However, their efficacy in high-stakes domains like finance is rarely explored, e.g., passing CFA exams and analyzing SEC filings. In this paper, we present the open-source FinLoRA project that benchmarks LoRA methods on both general and highly professional financial tasks. First, we curated 19 datasets covering diverse financial applications; in particular, we created four novel XBRL analysis datasets based on 150 SEC filings. Second, we evaluated five LoRA methods and five base LLMs. Finally, we provide extensive experimental results in terms of accuracy, F1, and BERTScore and report computational cost in terms of time and GPU memory during fine-tuning and inference stages. We find that LoRA methods achieved substantial performance gains of 36\% on average over base models. Our FinLoRA project provides an affordable and scalable approach to democratize financial intelligence to the general public. Datasets, LoRA adapters, code, and documentation are available at https://github.com/Open-Finance-Lab/FinLoRA

Paper Structure

This paper contains 33 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Is Fine-tuning of LLMs Needed on Financial Tasks?
  3. FinLoRA Benchmark
  4. Benchmark Tasks, Datasets, and Metrics
  5. Public Financial Datasets
  6. Newly-added XBRL Analysis Datasets
  7. Dataset Construction Pipeline
  8. Metrics
  9. Base Models and LoRA Methods
  10. Base Models
  11. LoRA Methods
  12. Benchmark Angles
  13. Angle I: LoRA Methods' Performance on Financial Datasets
  14. Angle II: LoRA Suitability for Financial Tasks
  15. Angle III: Resources of LoRA Fine-tuning and Inference
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Average performance of base models and LoRA models.
  • Figure 2: Task suitability.
  • Figure 3: Average inference time of LoRA fine-tuned Llama 3.1 8B and LoRA fine-tuned Gemini 2.0 FL across tasks