Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency

Toyin Aguda, Suchetha Siddagangappa, Elena Kochkina, Simerjot Kaur, Dongsheng Wang, Charese Smiley, Sameena Shah

TL;DR

This study evaluates three large language models (GPT-4, PaLM 2, and MPT Instruct) as domain-specific annotators for financial relation extraction on the REFinD dataset, comparing their outputs to expert and crowdworker annotations. It shows that GPT-4 and PaLM 2 generally outperform non-expert crowdworkers and offer substantial time and cost savings, though they do not reach expert-level accuracy. The paper introduces the LLM-RelIndex reliability metric to flag outputs requiring expert review and demonstrates how prompt design, including few-shot and chain-of-thought variants, influences performance, especially for smaller models. A hybrid annotation strategy combining automated LLM annotations with targeted expert intervention is recommended for scalable, finance-domain data labeling, aided by reliability signals to prioritize human review and reduce cost and time. The work provides practical guidance on prompt engineering, evaluation metrics, and cost analyses, highlighting both the promise and the current limits of LLMs as domain-specific data annotators in finance.

Abstract

Collecting labeled datasets in finance is challenging due to scarcity of domain experts and higher cost of employing them. While Large Language Models (LLMs) have demonstrated remarkable performance in data annotation tasks on general domain datasets, their effectiveness on domain specific datasets remains underexplored. To address this gap, we investigate the potential of LLMs as efficient data annotators for extracting relations in financial documents. We compare the annotations produced by three LLMs (GPT-4, PaLM 2, and MPT Instruct) against expert annotators and crowdworkers. We demonstrate that the current state-of-the-art LLMs can be sufficient alternatives to non-expert crowdworkers. We analyze models using various prompts and parameter settings and find that customizing the prompts for each relation group by providing specific examples belonging to those groups is paramount. Furthermore, we introduce a reliability index (LLM-RelIndex) used to identify outputs that may require expert attention. Finally, we perform an extensive time, cost and error analysis and provide recommendations for the collection and usage of automated annotations in domain-specific settings.

Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency

TL;DR

This study evaluates three large language models (GPT-4, PaLM 2, and MPT Instruct) as domain-specific annotators for financial relation extraction on the REFinD dataset, comparing their outputs to expert and crowdworker annotations. It shows that GPT-4 and PaLM 2 generally outperform non-expert crowdworkers and offer substantial time and cost savings, though they do not reach expert-level accuracy. The paper introduces the LLM-RelIndex reliability metric to flag outputs requiring expert review and demonstrates how prompt design, including few-shot and chain-of-thought variants, influences performance, especially for smaller models. A hybrid annotation strategy combining automated LLM annotations with targeted expert intervention is recommended for scalable, finance-domain data labeling, aided by reliability signals to prioritize human review and reduce cost and time. The work provides practical guidance on prompt engineering, evaluation metrics, and cost analyses, highlighting both the promise and the current limits of LLMs as domain-specific data annotators in finance.

Abstract

Collecting labeled datasets in finance is challenging due to scarcity of domain experts and higher cost of employing them. While Large Language Models (LLMs) have demonstrated remarkable performance in data annotation tasks on general domain datasets, their effectiveness on domain specific datasets remains underexplored. To address this gap, we investigate the potential of LLMs as efficient data annotators for extracting relations in financial documents. We compare the annotations produced by three LLMs (GPT-4, PaLM 2, and MPT Instruct) against expert annotators and crowdworkers. We demonstrate that the current state-of-the-art LLMs can be sufficient alternatives to non-expert crowdworkers. We analyze models using various prompts and parameter settings and find that customizing the prompts for each relation group by providing specific examples belonging to those groups is paramount. Furthermore, we introduce a reliability index (LLM-RelIndex) used to identify outputs that may require expert attention. Finally, we perform an extensive time, cost and error analysis and provide recommendations for the collection and usage of automated annotations in domain-specific settings.
Paper Structure (32 sections, 1 equation, 12 figures, 11 tables)

This paper contains 32 sections, 1 equation, 12 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset
  4. Experiments
  5. Models
  6. Evaluation
  7. Inter Annotator Agreement (IAA)
  8. Results
  9. Model Performance
  10. Inter-Annotator Agreement
  11. Error Analysis
  12. Semantic Ambiguity
  13. Relation Hallucinations
  14. Confident Misannotations
  15. LLM-RelIndex Based Accuracy Analysis
  16. ...and 17 more sections

Figures (12)

  • Figure 1: Example of relation extraction task from REFinD dataset.
  • Figure 2: Full instruction prompt example.
  • Figure 3: Annotator performance in terms of micro-averaged F1-Score under full instruction prompt.
  • Figure 4: Error Analysis: Qualitative examples illustrating different scenarios of how MTurk Crowdworkers and LLMs demonstrated high confidence on incorrect answer choices.
  • Figure 5: Human vs LLMs at Zero-shot using LLM-RelIndex
  • ...and 7 more figures