Benchmarking and Pushing the Multi-Bias Elimination Boundary of LLMs via Causal Effect Estimation-guided Debiasing

TL;DR

This work addresses the challenge that LLMs retain biases that impair generalization by introducing a multi-bias NLI benchmark in which every datum contains five biases with the same polarity. It shows that existing LLMs and debiasing methods falter under multi-bias conditions and that scaling alone is insufficient. The authors propose CMBE, a causal-effect estimation framework that estimates the average natural indirect effect of biases and subtracts it from the total causal effect to obtain the semantic direct effect, using a two-stage, linear-combination approach to handle multiple biases. Across multiple LLMs, CMBE improves generalization on the multi-bias benchmark while preserving in-domain performance, highlighting a promising route for robust inference in the presence of overlapping biases.

Abstract

Despite significant progress, recent studies have indicated that current large language models (LLMs) may still utilize bias during inference, leading to the poor generalizability of LLMs. Some benchmarks are proposed to investigate the generalizability of LLMs, with each piece of data typically containing one type of controlled bias. However, a single piece of data may contain multiple types of biases in practical applications. To bridge this gap, we propose a multi-bias benchmark where each piece of data contains five types of biases. The evaluations conducted on this benchmark reveal that the performance of existing LLMs and debiasing methods is unsatisfying, highlighting the challenge of eliminating multiple types of biases simultaneously. To overcome this challenge, we propose a causal effect estimation-guided multi-bias elimination method (CMBE). This method first estimates the causal effect of multiple types of biases simultaneously. Subsequently, we eliminate the causal effect of biases from the total causal effect exerted by both the semantic information and biases during inference. Experimental results show that CMBE can effectively eliminate multiple types of bias simultaneously to enhance the generalizability of LLMs.

Figures (5)

• Figure 1: This figure presents an example that contains five different types of bias.
• Figure 2: Error rates of four LLMs for each label.
• Figure 3: Causal Effect Estimation-guided Multi-bias Elimination Method.
• Figure 4: Influence of different numbers of data for estimating the average causal effects of different bias features combinations.
• Figure 5: Error rates of three LLMs for each label.