Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BUSTER: a "BUSiness Transaction Entity Recognition" dataset

Andrea Zugarini, Andrew Zamai, Marco Ernandes, Leonardo Rigutini

TL;DR

The paper addresses the mismatch between standard NLP benchmarks and real-world finance data by introducing BUSTER, a document-level business transaction ER benchmark comprising a gold corpus of 3779 annotated documents and a silver corpus of 6196 automatically annotated documents. It defines a tag-set focusing on entity roles in transactions (Parties, Advisors, Generic_Info) and evaluates four transformer models (BERT, RoBERTa, SEC-BERT, Longformer) under 5-fold cross-validation using strict token-level metrics and additional annotation quality measures. RoBERTa achieves the strongest overall performance, with Longformer performing comparably, while domain-specific SEC-BERT provides clear gains over vanilla BERT; the chunking strategy enables handling long financial documents. The dataset and protocols are publicly released to foster industry-oriented ER research in finance, with future work aiming to expand label coverage and introduce entity relations.

Abstract

Albeit Natural Language Processing has seen major breakthroughs in the last few years, transferring such advances into real-world business cases can be challenging. One of the reasons resides in the displacement between popular benchmarks and actual data. Lack of supervision, unbalanced classes, noisy data and long documents often affect real problems in vertical domains such as finance, law and health. To support industry-oriented research, we present BUSTER, a BUSiness Transaction Entity Recognition dataset. The dataset consists of 3779 manually annotated documents on financial transactions. We establish several baselines exploiting both general-purpose and domain-specific language models. The best performing model is also used to automatically annotate 6196 documents, which we release as an additional silver corpus to BUSTER.

BUSTER: a "BUSiness Transaction Entity Recognition" dataset

TL;DR

The paper addresses the mismatch between standard NLP benchmarks and real-world finance data by introducing BUSTER, a document-level business transaction ER benchmark comprising a gold corpus of 3779 annotated documents and a silver corpus of 6196 automatically annotated documents. It defines a tag-set focusing on entity roles in transactions (Parties, Advisors, Generic_Info) and evaluates four transformer models (BERT, RoBERTa, SEC-BERT, Longformer) under 5-fold cross-validation using strict token-level metrics and additional annotation quality measures. RoBERTa achieves the strongest overall performance, with Longformer performing comparably, while domain-specific SEC-BERT provides clear gains over vanilla BERT; the chunking strategy enables handling long financial documents. The dataset and protocols are publicly released to foster industry-oriented ER research in finance, with future work aiming to expand label coverage and introduce entity relations.

Abstract

Albeit Natural Language Processing has seen major breakthroughs in the last few years, transferring such advances into real-world business cases can be challenging. One of the reasons resides in the displacement between popular benchmarks and actual data. Lack of supervision, unbalanced classes, noisy data and long documents often affect real problems in vertical domains such as finance, law and health. To support industry-oriented research, we present BUSTER, a BUSiness Transaction Entity Recognition dataset. The dataset consists of 3779 manually annotated documents on financial transactions. We establish several baselines exploiting both general-purpose and domain-specific language models. The best performing model is also used to automatically annotate 6196 documents, which we release as an additional silver corpus to BUSTER.
Paper Structure (13 sections, 3 figures, 5 tables)

This paper contains 13 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Data Collection
  4. Annotation
  5. The BUSTER dataset
  6. Experiments
  7. Metrics.
  8. Dealing with long documents.
  9. BERT.
  10. RoBERTa.
  11. SEC-BERT.
  12. Longformer.
  13. Conclusions and future works

Figures (3)

  • Figure 1: An annotated example extracted from BUSTER.
  • Figure 2: Sequence length distribution of BUSTER documents in terms of words.
  • Figure 3: Distribution of tags families inside the documents.