Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Critical Questions Generation: Motivation and Challenges

Blanca Calvo Figueras, Rodrigo Agerri

TL;DR

Critical Questions Generation, consisting of processing an argumentative text to generate the critical questions (CQs) raised by it, is proposed, and it is concluded that, while LLMs are reasonable CQ generators, they still have a wide margin for improvement in this task.

Abstract

The development of Large Language Models (LLMs) has brought impressive performances on mitigation strategies against misinformation, such as counterargument generation. However, LLMs are still seriously hindered by outdated knowledge and by their tendency to generate hallucinated content. In order to circumvent these issues, we propose a new task, namely, Critical Questions Generation, consisting of processing an argumentative text to generate the critical questions (CQs) raised by it. In argumentation theory CQs are tools designed to lay bare the blind spots of an argument by pointing at the information it could be missing. Thus, instead of trying to deploy LLMs to produce knowledgeable and relevant counterarguments, we use them to question arguments, without requiring any external knowledge. Research on CQs Generation using LLMs requires a reference dataset for large scale experimentation. Thus, in this work we investigate two complementary methods to create such a resource: (i) instantiating CQs templates as defined by Walton's argumentation theory and (ii), using LLMs as CQs generators. By doing so, we contribute with a procedure to establish what is a valid CQ and conclude that, while LLMs are reasonable CQ generators, they still have a wide margin for improvement in this task.

Critical Questions Generation: Motivation and Challenges

TL;DR

Critical Questions Generation, consisting of processing an argumentative text to generate the critical questions (CQs) raised by it, is proposed, and it is concluded that, while LLMs are reasonable CQ generators, they still have a wide margin for improvement in this task.

Abstract

The development of Large Language Models (LLMs) has brought impressive performances on mitigation strategies against misinformation, such as counterargument generation. However, LLMs are still seriously hindered by outdated knowledge and by their tendency to generate hallucinated content. In order to circumvent these issues, we propose a new task, namely, Critical Questions Generation, consisting of processing an argumentative text to generate the critical questions (CQs) raised by it. In argumentation theory CQs are tools designed to lay bare the blind spots of an argument by pointing at the information it could be missing. Thus, instead of trying to deploy LLMs to produce knowledgeable and relevant counterarguments, we use them to question arguments, without requiring any external knowledge. Research on CQs Generation using LLMs requires a reference dataset for large scale experimentation. Thus, in this work we investigate two complementary methods to create such a resource: (i) instantiating CQs templates as defined by Walton's argumentation theory and (ii), using LLMs as CQs generators. By doing so, we contribute with a procedure to establish what is a valid CQ and conclude that, while LLMs are reasonable CQ generators, they still have a wide margin for improvement in this task.

Paper Structure

This paper contains 18 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Previous Work
  3. Using argumentation to fight misinformation
  4. The nature of critical questions
  5. Argumentation Schemes in Computational Argumentation
  6. Data
  7. Our Method
  8. Generation using theory
  9. Generation using LLMs
  10. Results
  11. Relevance with respect to the Intervention
  12. Relation of llm-CQs to Arguments and to theory-CQs
  13. Inferential Validity of the llm-CQs
  14. Comparing the Approaches
  15. Concluding Remarks
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Arguments from the US2016 dataset visser_annotating_2021, instantiated using the templates of argumentation schemes and critical questions defined in walton_argumentation_2008.
  • Figure 2: Outline of the steps taken in our approach. Starting from each intervention, we generate CQs using the theory templates (red-dotted box) and the LLMs (blue-dashed box). In the green box, we relate the relevant llm-CQs to the arguments of the intervention (if possible), and relate these llm-CQs to a theory-CQ (if possible).
  • Figure 3: Example of an instance of the generated reference data. The intervention is from the Moral Maze dataset, and the theory-CQs and the llm-CQs are the result of both of our generation methods.