Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is There a One-Model-Fits-All Approach to Information Extraction? Revisiting Task Definition Biases

Wenhao Huang, Qianyu He, Zhixu Li, Jiaqing Liang, Yanghua Xiao

TL;DR

To mitigate definition bias in information extraction, this work proposes a multi-stage framework consisting of definition bias measurement, bias-aware fine-tuning, and task-specific bias mitigation that demonstrates the effectiveness of the framework in addressing definition bias.

Abstract

Definition bias is a negative phenomenon that can mislead models. Definition bias in information extraction appears not only across datasets from different domains but also within datasets sharing the same domain. We identify two types of definition bias in IE: bias among information extraction datasets and bias between information extraction datasets and instruction tuning datasets. To systematically investigate definition bias, we conduct three probing experiments to quantitatively analyze it and discover the limitations of unified information extraction and large language models in solving definition bias. To mitigate definition bias in information extraction, we propose a multi-stage framework consisting of definition bias measurement, bias-aware fine-tuning, and task-specific bias mitigation. Experimental results demonstrate the effectiveness of our framework in addressing definition bias. Resources of this paper can be found at https://github.com/EZ-hwh/definition-bias

Is There a One-Model-Fits-All Approach to Information Extraction? Revisiting Task Definition Biases

TL;DR

To mitigate definition bias in information extraction, this work proposes a multi-stage framework consisting of definition bias measurement, bias-aware fine-tuning, and task-specific bias mitigation that demonstrates the effectiveness of the framework in addressing definition bias.

Abstract

Definition bias is a negative phenomenon that can mislead models. Definition bias in information extraction appears not only across datasets from different domains but also within datasets sharing the same domain. We identify two types of definition bias in IE: bias among information extraction datasets and bias between information extraction datasets and instruction tuning datasets. To systematically investigate definition bias, we conduct three probing experiments to quantitatively analyze it and discover the limitations of unified information extraction and large language models in solving definition bias. To mitigate definition bias in information extraction, we propose a multi-stage framework consisting of definition bias measurement, bias-aware fine-tuning, and task-specific bias mitigation. Experimental results demonstrate the effectiveness of our framework in addressing definition bias. Resources of this paper can be found at https://github.com/EZ-hwh/definition-bias
Paper Structure (37 sections, 6 equations, 4 figures, 16 tables)

This paper contains 37 sections, 6 equations, 4 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Definition Bias Probing Experiment
  3. Whether definition bias exists?
  4. Named Entity Recognition
  5. Relation Extraction
  6. Can UIE address definition bias?
  7. Source Prompt Tuning
  8. Source Prompt Inference
  9. Can LLMs address definition bias?
  10. Empirical Study of Definition Bias
  11. Whether definition bias exists?
  12. Can UIE address definition bias?
  13. Can LLM address definition bias?
  14. Alleviate Definition Bias
  15. Definition bias measurement
  16. ...and 22 more sections

Figures (4)

  • Figure 1: Definition bias among different datasets and LLMs even when they share the same entity type (for NER) or the same relation type (for RE).
  • Figure 2: Three settings for the probing tasks on definition bias across datasets, including (a) fully supervised, (b) source prompt and (c) LLMs zero/few-shot.
  • Figure 3: Our two-stage framework for alleviate definition bias. Left: we measure two kinds of definition bias with Fleiss' Kappa; Right: we first full-parameter fine-tune LLMs with measurement and then fine-tune with LoRA on specific dataset.
  • Figure 4: Ablation study on 12 information extraction dataset (NER and RE)