Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Potential of Large Language Models in Computational Argumentation

Guizhen Chen, Liying Cheng, Luu Anh Tuan, Lidong Bing

TL;DR

The paper assesses large language models on computational argumentation by organizing existing tasks into six categories across argument mining and generation, and by standardizing fourteen datasets. It introduces a novel counter speech generation benchmark to test end-to-end abilities in mining and generation, evaluated in zero-shot and few-shot settings across multiple LLMs. Experimental results show promising performance from GPT-3.5-Turbo and Flan-UL2, with Llama-2 models lagging in zero-shot contexts, and reveal that prompting strategies and model choice critically affect outcomes. The work provides benchmarks, prompts, and analyses that guide future evaluation of LLMs in computational argumentation and suggests directions to improve metrics and human-centered assessment for this domain.

Abstract

Computational argumentation has become an essential tool in various domains, including law, public policy, and artificial intelligence. It is an emerging research field in natural language processing that attracts increasing attention. Research on computational argumentation mainly involves two types of tasks: argument mining and argument generation. As large language models (LLMs) have demonstrated impressive capabilities in understanding context and generating natural language, it is worthwhile to evaluate the performance of LLMs on diverse computational argumentation tasks. This work aims to embark on an assessment of LLMs, such as ChatGPT, Flan models, and LLaMA2 models, in both zero-shot and few-shot settings. We organize existing tasks into six main categories and standardize the format of fourteen openly available datasets. In addition, we present a new benchmark dataset on counter speech generation that aims to holistically evaluate the end-to-end performance of LLMs on argument mining and argument generation. Extensive experiments show that LLMs exhibit commendable performance across most of the datasets, demonstrating their capabilities in the field of argumentation. Our analysis offers valuable suggestions for evaluating computational argumentation and its integration with LLMs in future research endeavors.

Exploring the Potential of Large Language Models in Computational Argumentation

TL;DR

The paper assesses large language models on computational argumentation by organizing existing tasks into six categories across argument mining and generation, and by standardizing fourteen datasets. It introduces a novel counter speech generation benchmark to test end-to-end abilities in mining and generation, evaluated in zero-shot and few-shot settings across multiple LLMs. Experimental results show promising performance from GPT-3.5-Turbo and Flan-UL2, with Llama-2 models lagging in zero-shot contexts, and reveal that prompting strategies and model choice critically affect outcomes. The work provides benchmarks, prompts, and analyses that guide future evaluation of LLMs in computational argumentation and suggests directions to improve metrics and human-centered assessment for this domain.

Abstract

Computational argumentation has become an essential tool in various domains, including law, public policy, and artificial intelligence. It is an emerging research field in natural language processing that attracts increasing attention. Research on computational argumentation mainly involves two types of tasks: argument mining and argument generation. As large language models (LLMs) have demonstrated impressive capabilities in understanding context and generating natural language, it is worthwhile to evaluate the performance of LLMs on diverse computational argumentation tasks. This work aims to embark on an assessment of LLMs, such as ChatGPT, Flan models, and LLaMA2 models, in both zero-shot and few-shot settings. We organize existing tasks into six main categories and standardize the format of fourteen openly available datasets. In addition, we present a new benchmark dataset on counter speech generation that aims to holistically evaluate the end-to-end performance of LLMs on argument mining and argument generation. Extensive experiments show that LLMs exhibit commendable performance across most of the datasets, demonstrating their capabilities in the field of argumentation. Our analysis offers valuable suggestions for evaluating computational argumentation and its integration with LLMs in future research endeavors.
Paper Structure (39 sections, 3 figures, 16 tables)

This paper contains 39 sections, 3 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Computational Argumentation
  4. Large Language Models
  5. Tasks and Datasets
  6. Argument Mining
  7. Claim Detection
  8. Evidence Detection
  9. Stance Detection
  10. Evidence Type Classification
  11. Argument Generation
  12. Generation
  13. Summarization
  14. Counter Speech Generation
  15. Experiments
  16. ...and 24 more sections

Figures (3)

  • Figure 1: Explored tasks and datasets in this work.
  • Figure 2: Three different approaches for our proposed task on counter speech generation.
  • Figure 3: Few-shot performance comparison on argument mining tasks. Results of the previous SOTA (using full setting and 500 samples) are also shown for easy comparison. *: Note that we exclude IBM Argument because the train set is smaller than 500.