Transfer Learning and Transformer Architecture for Financial Sentiment Analysis

TL;DR

The paper tackles the challenge of financial sentiment analysis under limited labeled data by leveraging transfer learning with BERT-style transformer architectures. It builds a pipeline that pre-trains on domain-specific corpora and fine-tunes with financial sentiment datasets, aiming to mitigate data scarcity and capture financial language nuances, including pandemic-related shifts. The study analyzes model depth, pretraining data sources, and catastrophic forgetting, reporting that fine-tuning on financial data improves sentiment classification while acknowledging practical constraints like model size and training time. The work highlights the potential of domain-adapted transformers for finance, with implications for continuous sentiment monitoring and real-time decision support, especially in pandemic contexts.

Abstract

Financial sentiment analysis allows financial institutions like Banks and Insurance Companies to better manage the credit scoring of their customers in a better way. Financial domain uses specialized mechanisms which makes sentiment analysis difficult. In this paper, we propose a pre-trained language model which can help to solve this problem with fewer labelled data. We extend on the principles of Transfer learning and Transformation architecture principles and also take into consideration recent outbreak of pandemics like COVID. We apply the sentiment analysis to two different sets of data. We also take smaller training set and fine tune the same as part of the model.

Figures (9)

• Figure 1: Masked LM Scenario
• Figure 2: Analyzing context of a sentence.
• Figure 3: Effect of BERT Pretraining
• Figure 4: BERT Process
• Figure 5: Configuration Data