Mitigating Selection Bias with Node Pruning and Auxiliary Options

TL;DR

This work tackles selection bias in MCQ answering by LLMs, a problem that undermines accuracy and reliability in decision-critical tasks. It introduces BNP, a parameter pruning method that removes bias-interacting rows in the final output projection, and AOI, a prompting strategy that adds an explicit 'I don't know' option to reduce bias even for black-box models; together, they yield improved accuracy and reduced bias across several LLMs and datasets. The authors also propose CKLD, a distribution-based metric that captures how closely the predicted answer distribution matches the ground-truth distribution, complementing existing bias measures like RSD. Results show BNP+AOI consistently improve performance and bias metrics, with notable gains (e.g., up to 24.9% accuracy improvement on ARC-Challenge for Llama-3) and strong generalization across white-box and black-box settings, suggesting practical applicability to diverse MCQ tasks and prompting paradigms.

Abstract

Large language models (LLMs) often exhibit systematic preferences for certain answer choices when responding to multiple-choice questions-a behavior known as selection bias. This bias reduces the accuracy and reliability of LLM outputs, limiting their usefulness in decision-critical applications. While prior work has focused on adjusting model inputs or outputs to mitigate this issue, our work takes a fundamentally different approach by identifying and removing the internal sources of bias. We introduce two methods: Bias Node Pruning (BNP), which prunes parameters that contribute to selection bias, and Auxiliary Option Injection (AOI), which introduces an additional answer choice to reduce bias in both white-box and black-box settings. To address the shortcomings of existing evaluation metrics, we propose Choice Kullback-Leibler Divergence (CKLD), a new metric that captures distributional imbalances in model predictions. Experiments on three LLMs across multiple datasets demonstrate that our methods consistently improve answer accuracy while reducing selection bias, providing a robust solution for both open- and closed-source models.

Figures (10)

• Figure 1: BNP and AOI reduce selection bias for white-box and black-box models. The CKLD metric is a standardized measure of selection bias.
• Figure 2: Comparison of the original and voting accuracy with different LLMs via zero-shot querying. Note, Claude3-Sonnet is evaluated under the black-box setting (§ \ref{['sec:blackbox']}).
• Figure 3: (a) Choice frequency tends to have a sharper distribution when the model's response is incorrect. (b) In Llama-3, selection bias is predominantly observed to be in the final output layer of the decoder. Other model figures are in Appendix \ref{['apdx:morefigs']}.
• Figure 4: Bias Node Pruning with Auxiliary Option Injection. (a) The bias vector $\mathbf{b}_\mathbf{x}$ is computed for each sample using its choice-permuted embeddings (Eq. \ref{['eq:biasvec']}). The bias vectors are averaged across a small subset of training data to retrieve the average bias vector, $\mathbf{b}$ (Eq. \ref{['eq:avgbiasvec']}). Then, $\mathbf{b}$ is used to select nodes to prune in $\bm{W}$, where $\bigotimes$ refers to the operation in Eq. \ref{['eq:topk']}. (b) The pruned $\tilde{\bm{W}}$ is used to retrieve answers for the test questions, along with our Auxiliary Option Injection technique that injects the "I don't know" option in the inputs (§ \ref{['sec:aoi']}). Our debiasing approaches may correct potentially erroneous responses retrieved with $\bm{W}$ and without AOI, as in (c).
• Figure 5: (a) BNP improves the base performances (dashed lines) regardless of the number of nodes pruned. The number of nodes to prune can be adjusted to achieve better performance. More figures are in Appendix \ref{['apdx:morefigs']}. (b) Each metric improvement ($\%$) from its base Llama-3 performance when using the average bias vector from different sources is shown in heatmaps.