L3iTC at the FinLLM Challenge Task: Quantization for Financial Text Classification & Summarization

TL;DR

This paper reports the L3iTC team's participation in the FinLLM Challenge 2024, focusing on financial text classification and summarization. The authors adopt a fine-tuning pipeline that combines 4-bit quantization with LoRA to reduce memory footprint and training time, enabling execution on low-GPU setups. They evaluate multiple Instruct LLMs (notably Mistral-7B variants and Meta-Llama-3-8B-Instruct) and find that a fine-tuned Mistral-7B-Inst-v0.3 model achieves 78% accuracy and 0.78 F1 on Task 1, securing 3rd place, while Task 2 results show a 6th-place finish with generation quality limited by quantization. The study highlights a trade-off between memory efficiency and generation quality, and suggests future work on higher-bit quantization and broader pretraining to enhance summarization performance while maintaining efficiency.

Abstract

This article details our participation (L3iTC) in the FinLLM Challenge Task 2024, focusing on two key areas: Task 1, financial text classification, and Task 2, financial text summarization. To address these challenges, we fine-tuned several large language models (LLMs) to optimize performance for each task. Specifically, we used 4-bit quantization and LoRA to determine which layers of the LLMs should be trained at a lower precision. This approach not only accelerated the fine-tuning process on the training data provided by the organizers but also enabled us to run the models on low GPU memory. Our fine-tuned models achieved third place for the financial classification task with an F1-score of 0.7543 and secured sixth place in the financial summarization task on the official test datasets.

Figures (1)

• Figure 1: Fine-tuning methodology