Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Annotation Guidelines-Based Knowledge Augmentation: Towards Enhancing Large Language Models for Educational Text Classification

Shiqi Liu, Sannyuya Liu, Lele Sha, Zijie Zeng, Dragan Gasevic, Zhi Liu

TL;DR

This work investigates how to improve large language models for educational text classification (LEC) by injecting label-definition knowledge from annotation guidelines through Annotation Guidelines-based Knowledge Augmentation (AGKA). The authors assemble a six-task LEC benchmark spanning behavior, emotion, and cognition, and evaluate a suite of non-fine-tuned and fine-tuned models with vanilla prompts, AGKA, and few-shot sampling via Random Under Sampler. Results show that AGKA significantly boosts non-fine-tuned LLMs (notably GPT-4.0 with AGKA few-shot) and, in some binary tasks, can surpass full-shot fine-tuned models; open-source Llama 3 70B with AGKA achieves performance on par with GPT-4.0 in several cases. However, multi-class tasks requiring deep semantic understanding (e.g., epistemic emotion, cognitive presence) remain challenging, and some LLMs can even degrade with AGKA in certain settings. Overall, AGKA offers a practical path to high-performance educational text classification with reduced annotation requirements, especially for open-source models, while highlighting the need for further research into complex semantic distinctions and ethical deployment in education.

Abstract

Various machine learning approaches have gained significant popularity for the automated classification of educational text to identify indicators of learning engagement -- i.e. learning engagement classification (LEC). LEC can offer comprehensive insights into human learning processes, attracting significant interest from diverse research communities, including Natural Language Processing (NLP), Learning Analytics, and Educational Data Mining. Recently, Large Language Models (LLMs), such as ChatGPT, have demonstrated remarkable performance in various NLP tasks. However, their comprehensive evaluation and improvement approaches in LEC tasks have not been thoroughly investigated. In this study, we propose the Annotation Guidelines-based Knowledge Augmentation (AGKA) approach to improve LLMs. AGKA employs GPT 4.0 to retrieve label definition knowledge from annotation guidelines, and then applies the random under-sampler to select a few typical examples. Subsequently, we conduct a systematic evaluation benchmark of LEC, which includes six LEC datasets covering behavior classification (question and urgency level), emotion classification (binary and epistemic emotion), and cognition classification (opinion and cognitive presence). The study results demonstrate that AGKA can enhance non-fine-tuned LLMs, particularly GPT 4.0 and Llama 3 70B. GPT 4.0 with AGKA few-shot outperforms full-shot fine-tuned models such as BERT and RoBERTa on simple binary classification datasets. However, GPT 4.0 lags in multi-class tasks that require a deep understanding of complex semantic information. Notably, Llama 3 70B with AGKA is a promising combination based on open-source LLM, because its performance is on par with closed-source GPT 4.0 with AGKA. In addition, LLMs struggle to distinguish between labels with similar names in multi-class classification.

Annotation Guidelines-Based Knowledge Augmentation: Towards Enhancing Large Language Models for Educational Text Classification

TL;DR

This work investigates how to improve large language models for educational text classification (LEC) by injecting label-definition knowledge from annotation guidelines through Annotation Guidelines-based Knowledge Augmentation (AGKA). The authors assemble a six-task LEC benchmark spanning behavior, emotion, and cognition, and evaluate a suite of non-fine-tuned and fine-tuned models with vanilla prompts, AGKA, and few-shot sampling via Random Under Sampler. Results show that AGKA significantly boosts non-fine-tuned LLMs (notably GPT-4.0 with AGKA few-shot) and, in some binary tasks, can surpass full-shot fine-tuned models; open-source Llama 3 70B with AGKA achieves performance on par with GPT-4.0 in several cases. However, multi-class tasks requiring deep semantic understanding (e.g., epistemic emotion, cognitive presence) remain challenging, and some LLMs can even degrade with AGKA in certain settings. Overall, AGKA offers a practical path to high-performance educational text classification with reduced annotation requirements, especially for open-source models, while highlighting the need for further research into complex semantic distinctions and ethical deployment in education.

Abstract

Various machine learning approaches have gained significant popularity for the automated classification of educational text to identify indicators of learning engagement -- i.e. learning engagement classification (LEC). LEC can offer comprehensive insights into human learning processes, attracting significant interest from diverse research communities, including Natural Language Processing (NLP), Learning Analytics, and Educational Data Mining. Recently, Large Language Models (LLMs), such as ChatGPT, have demonstrated remarkable performance in various NLP tasks. However, their comprehensive evaluation and improvement approaches in LEC tasks have not been thoroughly investigated. In this study, we propose the Annotation Guidelines-based Knowledge Augmentation (AGKA) approach to improve LLMs. AGKA employs GPT 4.0 to retrieve label definition knowledge from annotation guidelines, and then applies the random under-sampler to select a few typical examples. Subsequently, we conduct a systematic evaluation benchmark of LEC, which includes six LEC datasets covering behavior classification (question and urgency level), emotion classification (binary and epistemic emotion), and cognition classification (opinion and cognitive presence). The study results demonstrate that AGKA can enhance non-fine-tuned LLMs, particularly GPT 4.0 and Llama 3 70B. GPT 4.0 with AGKA few-shot outperforms full-shot fine-tuned models such as BERT and RoBERTa on simple binary classification datasets. However, GPT 4.0 lags in multi-class tasks that require a deep understanding of complex semantic information. Notably, Llama 3 70B with AGKA is a promising combination based on open-source LLM, because its performance is on par with closed-source GPT 4.0 with AGKA. In addition, LLMs struggle to distinguish between labels with similar names in multi-class classification.
Paper Structure (22 sections, 1 equation, 4 figures, 4 tables)

This paper contains 22 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Learning Engagement Classification
  4. Large Language Models in LEC
  5. Prompt Engineering
  6. Method
  7. LEC Benchmark Datasets
  8. Annotation Guidelines-based Knowledge Augmentation (AGKA)
  9. Problem Definition
  10. Components of AGKA
  11. Prompt Baseline
  12. Model Selection
  13. Evaluation
  14. Results
  15. Overall Results Analysis
  16. ...and 7 more sections

Figures (4)

  • Figure 1: The universal process of LEC based on LLMs with AGKA.
  • Figure 2: Components of AGKA. AGKA = Task claim + label definition knowledge + Output format definition + (Typical few-shot) + The text to be predicted. Vanilla = Task claim + Output format definition + The text to be predicted. RUS refers to Random Under Sampler.
  • Figure 3: Results in few-shot setting of fine-tune models and non fine-tune LLMs. BC, EC and CC refer to the Classification of Behavior, Emotion and Cognition.
  • Figure 4: The confusion matrix of the best prediction results from the GPT series LLMs