Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extraction

TL;DR

The approach effectively alleviates prompt bias in knowledge evaluation, thereby enhancing the reliability of benchmark assessments, and can be a golden standard to strengthen PLMs toward reliable knowledge bases.

Abstract

Recent research shows that pre-trained language models (PLMs) suffer from "prompt bias" in factual knowledge extraction, i.e., prompts tend to introduce biases toward specific labels. Prompt bias presents a significant challenge in assessing the factual knowledge within PLMs. Therefore, this paper aims to improve the reliability of existing benchmarks by thoroughly investigating and mitigating prompt bias. We show that: 1) all prompts in the experiments exhibit non-negligible bias, with gradient-based prompts like AutoPrompt and OptiPrompt displaying significantly higher levels of bias; 2) prompt bias can amplify benchmark accuracy unreasonably by overfitting the test datasets, especially on imbalanced datasets like LAMA. Based on these findings, we propose a representation-based approach to mitigate the prompt bias during inference time. Specifically, we first estimate the biased representation using prompt-only querying, and then remove it from the model's internal representations to generate the debiased representations, which are used to produce the final debiased outputs. Experiments across various prompts, PLMs, and benchmarks show that our approach can not only correct the overfitted performance caused by prompt bias, but also significantly improve the prompt retrieval capability (up to 10% absolute performance gain). These results indicate that our approach effectively alleviates prompt bias in knowledge evaluation, thereby enhancing the reliability of benchmark assessments. Hopefully, our plug-and-play approach can be a golden standard to strengthen PLMs toward reliable knowledge bases. Code and data are released in https://github.com/FelliYang/PromptBias.

Figures (8)

• Figure 1: Language models surfer from unintended prompt bias in factual knowledge extraction. When querying BERT the religion of Albanians, the model is affected by the prompt bias and makes an incorrect prediction Christian. With our debiasing approach, the model rectifies its prediction to the correct answer Muslim.
• Figure 2: Examples of prompt bias from LAMA manual prompts. "Prompt bias" shows the BERT-base model probability distributions probed using prompt-only querying, while "uniform" shows an ideal unbiased distribution for reference. Prompts are biased towards certain labels.
• Figure 3: Quantified prompt bias for various prompts and PLMs using J-S divergence, averaged on 41 relations in the LAMA benchmark.
• Figure 4: The pipeline of our method. The red line represents the process of debiasing, which uses the subtraction of the representations between prompt-only query $V_{T(\cdot)}$ and vanilla query $V_{T(x)}$.
• Figure 5: The ratio of biased data in the performance degraded by debiasing, across diverse PLMs and prompts, averaged on 41 relations. Performance degradation mostly comes from biased data.