Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Option-ID Based Elimination For Multiple Choice Questions

Zhenhao Zhu, Bulou Liu, Qingyao Ai, Yiqun Liu

TL;DR

This work tackles improving large language models' performance on multiple-choice questions by reframing option selection as a PoE problem over option IDs. It introduces PoE_ID, a debiased elimination framework that uses Pride-based debiasing to counter token bias, with two elimination strategies: log-threshold and sequential elimination. Across six open-source LLMs and four diverse MCQ datasets, PoE_ID, especially the log-based variant, yields robust zero-shot and few-shot improvements, with larger gains on questions with more options. Analyses show that elimination reduces noise and increases the probability mass of the correct option's ID after refinement, and that replacing options with [MASK] is generally inferior to direct elimination. The work also uncovers intrinsic limitations in LLMs’ ability to explicitly identify incorrect options, motivating future exploration of prompts and prompting strategies to further enhance MCQ reasoning in LLMs.

Abstract

Multiple choice questions (MCQs) are a popular and important task for evaluating large language models (LLMs). Based on common strategies people use when answering MCQs, the process of elimination (PoE) has been proposed as an effective problem-solving method. Existing PoE methods typically either have LLMs directly identify incorrect options or score options and replace lower-scoring ones with [MASK]. However, both methods suffer from inapplicability or suboptimal performance. To address these issues, this paper proposes a novel option-ID based PoE ($\text{PoE}_{\text{ID}}$). $\text{PoE}_{\text{ID}}$ critically incorporates a debiasing technique to counteract LLMs token bias, enhancing robustness over naive ID-based elimination. It features two strategies: $\text{PoE}_{\text{ID}}^{\text{log}}$, which eliminates options whose IDs have log probabilities below the average threshold, and $\text{PoE}_{\text{ID}}^{\text{seq}}$, which iteratively removes the option with the lowest ID probability. We conduct extensive experiments with 6 different LLMs on 4 diverse datasets. The results demonstrate that $\text{PoE}_{\text{ID}}$, especially $\text{PoE}_{\text{ID}}^{\text{log}}$, significantly improves zero-shot and few-shot MCQs performance, particularly in datasets with more options. Our analyses demonstrate that $\text{PoE}_{\text{ID}}^{\text{log}}$ enhances the LLMs' confidence in selecting the correct option, and the option elimination strategy outperforms methods relying on [MASK] replacement. We further investigate the limitations of LLMs in directly identifying incorrect options, which stem from their inherent deficiencies.

Option-ID Based Elimination For Multiple Choice Questions

TL;DR

This work tackles improving large language models' performance on multiple-choice questions by reframing option selection as a PoE problem over option IDs. It introduces PoE_ID, a debiased elimination framework that uses Pride-based debiasing to counter token bias, with two elimination strategies: log-threshold and sequential elimination. Across six open-source LLMs and four diverse MCQ datasets, PoE_ID, especially the log-based variant, yields robust zero-shot and few-shot improvements, with larger gains on questions with more options. Analyses show that elimination reduces noise and increases the probability mass of the correct option's ID after refinement, and that replacing options with [MASK] is generally inferior to direct elimination. The work also uncovers intrinsic limitations in LLMs’ ability to explicitly identify incorrect options, motivating future exploration of prompts and prompting strategies to further enhance MCQ reasoning in LLMs.

Abstract

Multiple choice questions (MCQs) are a popular and important task for evaluating large language models (LLMs). Based on common strategies people use when answering MCQs, the process of elimination (PoE) has been proposed as an effective problem-solving method. Existing PoE methods typically either have LLMs directly identify incorrect options or score options and replace lower-scoring ones with [MASK]. However, both methods suffer from inapplicability or suboptimal performance. To address these issues, this paper proposes a novel option-ID based PoE (). critically incorporates a debiasing technique to counteract LLMs token bias, enhancing robustness over naive ID-based elimination. It features two strategies: , which eliminates options whose IDs have log probabilities below the average threshold, and , which iteratively removes the option with the lowest ID probability. We conduct extensive experiments with 6 different LLMs on 4 diverse datasets. The results demonstrate that , especially , significantly improves zero-shot and few-shot MCQs performance, particularly in datasets with more options. Our analyses demonstrate that enhances the LLMs' confidence in selecting the correct option, and the option elimination strategy outperforms methods relying on [MASK] replacement. We further investigate the limitations of LLMs in directly identifying incorrect options, which stem from their inherent deficiencies.
Paper Structure (31 sections, 7 equations, 2 figures, 10 tables)

This paper contains 31 sections, 7 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs
  4. MCQs
  5. Methodology
  6. Problem Setting
  7. PoE
  8. Debias
  9. Elimination Strategies
  10. Sequential Elimination ($\text{PoE}_{\text{ID}}^{\textbf{seq}}$)
  11. Eliminating Options Below Average Log-Probability ($\text{PoE}_{\text{ID}}^{\textbf{log}}$)
  12. Experimental Setup
  13. Datasets
  14. LLM Selection and Configuration
  15. Baselines
  16. ...and 16 more sections

Figures (2)

  • Figure 1: An illustration of $\text{PoE}_{\text{ID}}^{\text{log}}$ for a multiple choice question. The log probabilities of IDs B and D fall below the average threshold, leading to options B and D being eliminated from the option space.
  • Figure 2: The blue bars represent the proportion of samples where the probability of the correct option's ID increased after the elimination process, while the orange bars show the average magnitude of this probability increase.