Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Comprehensive Argument Analysis in Education: Dataset, Tasks, and Method

Yupei Ren, Xinyi Zhou, Ning Zhang, Shangqing Zhao, Man Lan, Xiaopeng Bai

TL;DR

The paper addresses the need for comprehensive argument analysis in education by introducing a CEAMC-based corpus enriched with 14 fine-grained vertical and horizontal argument relation types. It develops a multi-task framework for Argument Component Detection, Relation Prediction, and Automated Essay Grading, backed by a detailed annotation scheme and reliability analysis. Through experiments with PLMs, LoRA-tuned Chinese LLMs, and GPT-4 across tasks, it demonstrates that task type and domain significantly influence model performance and that fine-grained annotations enhance essay evaluation. The work highlights the value of multi-dimensional argument analysis for educational assessment and provides benchmarks and insights for future research, while acknowledging limitations in data scale and annotation subjectivity.

Abstract

Argument mining has garnered increasing attention over the years, with the recent advancement of Large Language Models (LLMs) further propelling this trend. However, current argument relations remain relatively simplistic and foundational, struggling to capture the full scope of argument information, particularly when it comes to representing complex argument structures in real-world scenarios. To address this limitation, we propose 14 fine-grained relation types from both vertical and horizontal dimensions, thereby capturing the intricate interplay between argument components for a thorough understanding of argument structure. On this basis, we conducted extensive experiments on three tasks: argument component detection, relation prediction, and automated essay grading. Additionally, we explored the impact of writing quality on argument component detection and relation prediction, as well as the connections between discourse relations and argumentative features. The findings highlight the importance of fine-grained argumentative annotations for argumentative writing quality assessment and encourage multi-dimensional argument analysis.

Towards Comprehensive Argument Analysis in Education: Dataset, Tasks, and Method

TL;DR

The paper addresses the need for comprehensive argument analysis in education by introducing a CEAMC-based corpus enriched with 14 fine-grained vertical and horizontal argument relation types. It develops a multi-task framework for Argument Component Detection, Relation Prediction, and Automated Essay Grading, backed by a detailed annotation scheme and reliability analysis. Through experiments with PLMs, LoRA-tuned Chinese LLMs, and GPT-4 across tasks, it demonstrates that task type and domain significantly influence model performance and that fine-grained annotations enhance essay evaluation. The work highlights the value of multi-dimensional argument analysis for educational assessment and provides benchmarks and insights for future research, while acknowledging limitations in data scale and annotation subjectivity.

Abstract

Argument mining has garnered increasing attention over the years, with the recent advancement of Large Language Models (LLMs) further propelling this trend. However, current argument relations remain relatively simplistic and foundational, struggling to capture the full scope of argument information, particularly when it comes to representing complex argument structures in real-world scenarios. To address this limitation, we propose 14 fine-grained relation types from both vertical and horizontal dimensions, thereby capturing the intricate interplay between argument components for a thorough understanding of argument structure. On this basis, we conducted extensive experiments on three tasks: argument component detection, relation prediction, and automated essay grading. Additionally, we explored the impact of writing quality on argument component detection and relation prediction, as well as the connections between discourse relations and argumentative features. The findings highlight the importance of fine-grained argumentative annotations for argumentative writing quality assessment and encourage multi-dimensional argument analysis.
Paper Structure (30 sections, 3 figures, 9 tables)

This paper contains 30 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Argument Mining
  4. Discourse Relation Recognition
  5. Corpus Construction
  6. Source Data
  7. Annotation Scheme
  8. Vertical Dimension
  9. Stance-Based Argument Relations
  10. Evidence-Based Argument Relations
  11. Discourse-Based Argument Relations
  12. Horizontal Dimension
  13. Annotation Process
  14. Inner Annotator Agreements
  15. Data Statistics
  16. ...and 15 more sections

Figures (3)

  • Figure 1: An Annotation Example (Excerpt). The red font indicates argument component types, the blue arrows on the right signify vertical argument relations, and the green arrow on the left represent horizontal logical relations. The content above the arrows corresponds to the respective relation types.
  • Figure 2: Effect of negative sampling for Relation Prediction task with RoBERTa and ChatGLM models.
  • Figure 3: ENA networks of discourse and argument relations in high- (red) and low-quality essays (blue).