Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In-Context Learning and Fine-Tuning GPT for Argument Mining

Jérémie Cabessa, Hugo Hernault, Umer Mushtaq

TL;DR

This work investigates both training-free and fine-tuning approaches for Argument Type Classification (ATC) within Argument Mining. It proposes a 2-step In-Context Learning (ICL) strategy that uses $k \in \{3,5\}$ neighbor selection and $n \in \{3,5\}$-fold ensembling to solve ATC with GPT-4 in a training-free setting, achieving competitive macro-F1 scores by leveraging discursive context from demonstrations. In parallel, it fine-tunes GPT-3.5-turbo with carefully crafted textual and structural features, attaining state-of-the-art macro-F1 on ATC. The results highlight emergent capabilities of large language models to capture global argumentative flow, with ICL offering robust training-free performance and fine-tuning providing the best task-specific accuracy. The work also provides a GitHub implementation to facilitate replication and extension to other Argument Mining sub-tasks.

Abstract

Large Language Models (LLMs) have become ubiquitous in NLP and deep learning. In-Context Learning (ICL) has been suggested as a bridging paradigm between the training-free and fine-tuning LLMs settings. In ICL, an LLM is conditioned to solve tasks by means of a few solved demonstration examples included as prompt. Argument Mining (AM) aims to extract the complex argumentative structure of a text, and Argument Type Classification (ATC) is an essential sub-task of AM. We introduce an ICL strategy for ATC combining kNN-based examples selection and majority vote ensembling. In the training-free ICL setting, we show that GPT-4 is able to leverage relevant information from only a few demonstration examples and achieve very competitive classification accuracy on ATC. We further set up a fine-tuning strategy incorporating well-crafted structural features given directly in textual form. In this setting, GPT-3.5 achieves state-of-the-art performance on ATC. Overall, these results emphasize the emergent ability of LLMs to grasp global discursive flow in raw text in both off-the-shelf and fine-tuned setups.

In-Context Learning and Fine-Tuning GPT for Argument Mining

TL;DR

This work investigates both training-free and fine-tuning approaches for Argument Type Classification (ATC) within Argument Mining. It proposes a 2-step In-Context Learning (ICL) strategy that uses neighbor selection and -fold ensembling to solve ATC with GPT-4 in a training-free setting, achieving competitive macro-F1 scores by leveraging discursive context from demonstrations. In parallel, it fine-tunes GPT-3.5-turbo with carefully crafted textual and structural features, attaining state-of-the-art macro-F1 on ATC. The results highlight emergent capabilities of large language models to capture global argumentative flow, with ICL offering robust training-free performance and fine-tuning providing the best task-specific accuracy. The work also provides a GitHub implementation to facilitate replication and extension to other Argument Mining sub-tasks.

Abstract

Large Language Models (LLMs) have become ubiquitous in NLP and deep learning. In-Context Learning (ICL) has been suggested as a bridging paradigm between the training-free and fine-tuning LLMs settings. In ICL, an LLM is conditioned to solve tasks by means of a few solved demonstration examples included as prompt. Argument Mining (AM) aims to extract the complex argumentative structure of a text, and Argument Type Classification (ATC) is an essential sub-task of AM. We introduce an ICL strategy for ATC combining kNN-based examples selection and majority vote ensembling. In the training-free ICL setting, we show that GPT-4 is able to leverage relevant information from only a few demonstration examples and achieve very competitive classification accuracy on ATC. We further set up a fine-tuning strategy incorporating well-crafted structural features given directly in textual form. In this setting, GPT-3.5 achieves state-of-the-art performance on ATC. Overall, these results emphasize the emergent ability of LLMs to grasp global discursive flow in raw text in both off-the-shelf and fine-tuned setups.
Paper Structure (8 sections, 1 figure, 3 tables)

This paper contains 8 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Dataset
  4. In-Context Learning (ICL)
  5. Additional information and features
  6. Fine-tuning
  7. Results
  8. Conclusion

Figures (1)

  • Figure 1: 2-step ICL approach: a $k$NN-based example prediction ($k=3, 5$) step followed by an $n$-Ensembling ($n=3, 5$) step (cf. text for further details).