Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models

Jaeyoung Choe, Keonwoong Noh, Nayeon Kim, Seyun Ahn, Woohwan Jung

TL;DR

The paper identifies a key limitation of financial pretrained language models: narrow pretraining data diversity which constrains generalization across diverse financial tasks. It introduces FiLM, a RoBERTa-base encoder pretrained on a broad, five-group financial corpus using a one-epoch masked language modeling objective, after careful preprocessing including deduplication. FiLM demonstrates strong, robust performance across six financial downstream tasks, often surpassing both existing financial PLMs and general-domain models, and even outperforming on macroeconomic text like FOMC; importantly, diversity enables stronger generalization to unseen corpora while reducing training energy. These results underscore the value of diverse domain data for improving the practicality and sustainability of domain-specific language models, and they provide publicly available resources to advance financial NLP broadly.

Abstract

Over the past few years, various domain-specific pretrained language models (PLMs) have been proposed and have outperformed general-domain PLMs in specialized areas such as biomedical, scientific, and clinical domains. In addition, financial PLMs have been studied because of the high economic impact of financial data analysis. However, we found that financial PLMs were not pretrained on sufficiently diverse financial data. This lack of diverse training data leads to a subpar generalization performance, resulting in general-purpose PLMs, including BERT, often outperforming financial PLMs on many downstream tasks. To address this issue, we collected a broad range of financial corpus and trained the Financial Language Model (FiLM) on these diverse datasets. Our experimental results confirm that FiLM outperforms not only existing financial PLMs but also general domain PLMs. Furthermore, we provide empirical evidence that this improvement can be achieved even for unseen corpus groups.

Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models

TL;DR

The paper identifies a key limitation of financial pretrained language models: narrow pretraining data diversity which constrains generalization across diverse financial tasks. It introduces FiLM, a RoBERTa-base encoder pretrained on a broad, five-group financial corpus using a one-epoch masked language modeling objective, after careful preprocessing including deduplication. FiLM demonstrates strong, robust performance across six financial downstream tasks, often surpassing both existing financial PLMs and general-domain models, and even outperforming on macroeconomic text like FOMC; importantly, diversity enables stronger generalization to unseen corpora while reducing training energy. These results underscore the value of diverse domain data for improving the practicality and sustainability of domain-specific language models, and they provide publicly available resources to advance financial NLP broadly.

Abstract

Over the past few years, various domain-specific pretrained language models (PLMs) have been proposed and have outperformed general-domain PLMs in specialized areas such as biomedical, scientific, and clinical domains. In addition, financial PLMs have been studied because of the high economic impact of financial data analysis. However, we found that financial PLMs were not pretrained on sufficiently diverse financial data. This lack of diverse training data leads to a subpar generalization performance, resulting in general-purpose PLMs, including BERT, often outperforming financial PLMs on many downstream tasks. To address this issue, we collected a broad range of financial corpus and trained the Financial Language Model (FiLM) on these diverse datasets. Our experimental results confirm that FiLM outperforms not only existing financial PLMs but also general domain PLMs. Furthermore, we provide empirical evidence that this improvement can be achieved even for unseen corpus groups.
Paper Structure (25 sections, 4 figures, 14 tables)

This paper contains 25 sections, 4 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Proposed method
  3. Pretraining datasets
  4. Preprocessing
  5. Training procedure for FiLM
  6. Financial tasks for evaluation
  7. Experiments
  8. Experiment setup
  9. Main results
  10. Impacts of the diversity in the pretraining dataset
  11. Performance extrapolation
  12. SEC filings dataset
  13. Macroeconomic perspective
  14. Cost-efficiency
  15. Pretraining information
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Sentence embeddings visualization for both corpus groups and financial tasks.
  • Figure 2: Average F1 scores measured on four financial tasks, with varying the number of corpus groups for pretraining.
  • Figure 3: Vocabulary overlap ratio between pretraining and downstream task datasets.
  • Figure 4: Two nearest corpus groups to each downstream dataset in embedding space.