'Finance Wizard' at the FinLLM Challenge Task: Financial Text Summarization

TL;DR

The paper tackles financial text summarization by building a task-specific FinLLM through a three-stage pipeline: continual pre-training on finance-related corpora, multi-task instruction-tuning on finance benchmarks, and final task-specific instruction-tuning to create FinLlama3_sum. It selects Llama3 8B for its large context window, trains it on diverse finance data, and demonstrates progressive gains culminating in a ROUGE-1 score of $0.521$ and a 3rd-place finish. Compared with baseline foundation models and multi-task FinLLMs, the results show that task-specific tuning yields substantial improvements over generalist or multi-task approaches, while also highlighting issues like potential catastrophic forgetting. The work highlights practical implications for building domain-specialized summarizers and provides open-access resources (model, code, and datasets) to facilitate further research and benchmarking, with future plans to scale to larger models and broaden evaluation through FinBen benchmarks.

Abstract

This paper presents our participation under the team name `Finance Wizard' in the FinNLP-AgentScen 2024 shared task #2: Financial Text Summarization. It documents our pipeline approach of fine-tuning a foundation model into a task-specific model for Financial Text Summarization. It involves (1) adapting Llama3 8B, a foundation model, to the Finance domain via continued pre-training, (2) multi-task instruction-tuning to further equip the model with more finance-related capabilities, (3) finally fine-tuning the model into a task-specific `expert'. Our model, FinLlama3\_sum, yielded commendable results, securing the third position in its category with a ROUGE-1 score of 0.521.

Figures (1)

• Figure 1: The design of our end-to-end fine-tuning approach. This shows the evolution of a foundation model to the final task-specific 'expert' for financial text summarization.