Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empirical Study of Mutual Reinforcement Effect and Application in Few-shot Text Classification Tasks via Prompt

Chengguang Gan, Tatsunori Mori

TL;DR

The paper investigates the Mutual Reinforcement Effect (MRE), a bidirectional enhancement between word-level information extraction and text-level classification. It introduces novel input-output ablations to empirically validate MRE across 21 Multilingual MRE Mix datasets and extends the concept to prompt-based few-shot learning using word-level information as verbalizers. Results demonstrate consistent MRE evidence, with word-level information boosting text-level predictions and vice versa, and show that Word-Level Information–Knowledgeable Verbalizers outperform traditional verbalizers in most datasets, especially for sentiment tasks. These findings provide a practical framework for leveraging cross-level cues in multilingual text classification and information extraction using large language models, with implications for robust prompt design and few-shot learning.

Abstract

The Mutual Reinforcement Effect (MRE) investigates the synergistic relationship between word-level and text-level classifications in text classification tasks. It posits that the performance of both classification levels can be mutually enhanced. However, this mechanism has not been adequately demonstrated or explained in prior research. To address this gap, we employ empirical experiment to observe and substantiate the MRE theory. Our experiments on 21 MRE mix datasets revealed the presence of MRE in the model and its impact. Specifically, we conducted compare experiments use fine-tune. The results of findings from comparison experiments corroborates the existence of MRE. Furthermore, we extended the application of MRE to prompt learning, utilizing word-level information as a verbalizer to bolster the model's prediction of text-level classification labels. In our final experiment, the F1-score significantly surpassed the baseline in 18 out of 21 MRE Mix datasets, further validating the notion that word-level information enhances the language model's comprehension of the text as a whole.

Empirical Study of Mutual Reinforcement Effect and Application in Few-shot Text Classification Tasks via Prompt

TL;DR

The paper investigates the Mutual Reinforcement Effect (MRE), a bidirectional enhancement between word-level information extraction and text-level classification. It introduces novel input-output ablations to empirically validate MRE across 21 Multilingual MRE Mix datasets and extends the concept to prompt-based few-shot learning using word-level information as verbalizers. Results demonstrate consistent MRE evidence, with word-level information boosting text-level predictions and vice versa, and show that Word-Level Information–Knowledgeable Verbalizers outperform traditional verbalizers in most datasets, especially for sentiment tasks. These findings provide a practical framework for leveraging cross-level cues in multilingual text classification and information extraction using large language models, with implications for robust prompt design and few-shot learning.

Abstract

The Mutual Reinforcement Effect (MRE) investigates the synergistic relationship between word-level and text-level classifications in text classification tasks. It posits that the performance of both classification levels can be mutually enhanced. However, this mechanism has not been adequately demonstrated or explained in prior research. To address this gap, we employ empirical experiment to observe and substantiate the MRE theory. Our experiments on 21 MRE mix datasets revealed the presence of MRE in the model and its impact. Specifically, we conducted compare experiments use fine-tune. The results of findings from comparison experiments corroborates the existence of MRE. Furthermore, we extended the application of MRE to prompt learning, utilizing word-level information as a verbalizer to bolster the model's prediction of text-level classification labels. In our final experiment, the F1-score significantly surpassed the baseline in 18 out of 21 MRE Mix datasets, further validating the notion that word-level information enhances the language model's comprehension of the text as a whole.

Paper Structure

This paper contains 12 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Empirical Experiment of Mutual Reinforcement Effect
  4. Word-level Information as Knowledgeable Verbalizer
  5. Experiment Setup
  6. Datasets
  7. Model
  8. Results
  9. Empirical Experiment of Mutual Reinforcement Effect
  10. Word-level Information as Knowledgeable Verbalizer
  11. Conclusion
  12. Limitation

Figures (4)

  • Figure 1: The figure illustrates the mutual reinforcement effect between text-level and word-level tasks in sentiment classification task.
  • Figure 2: The figure shows the inputs and outputs of the traditional ablation experiment for the MRE task and the new empirical MRE experiment proposed in this work.
  • Figure 3: The figure illustrates the flow of an empirical MRE experiment using the new approach.
  • Figure 4: The figure demonstrates how word-level information is utilized as a Knowledgeable Verbalizer to assist in text-level classification tasks. Additionally, it provides a detailed explanation of the functioning of the Knowledgeable Verbalizer.