Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoning

Kiran Purohit, Venktesh V, Raghuram Devalla, Krishna Mohan Yerragorla, Sourangshu Bhattacharya, Avishek Anand

TL;DR

ExPLORA is introduced, a novel exploration method designed to estimate the parameters of the scoring function, which evaluates exemplar subsets without incorporating confidence information, and significantly reduces the number of LLM calls to ~11% of those required by state-of-the-art methods and achieves a substantial performance improvement of 12.24%.

Abstract

Answering reasoning-based complex questions over text and hybrid sources, including tables, is a challenging task. Recent advances in large language models (LLMs) have enabled in-context learning (ICL), allowing LLMs to acquire proficiency in a specific task using only a few demonstration samples (exemplars). A critical challenge in ICL is the selection of optimal exemplars, which can be either task-specific (static) or test-example-specific (dynamic). Static exemplars provide faster inference times and increased robustness across a distribution of test examples. In this paper, we propose an algorithm for static exemplar subset selection for complex reasoning tasks. We introduce EXPLORA, a novel exploration method designed to estimate the parameters of the scoring function, which evaluates exemplar subsets without incorporating confidence information. EXPLORA significantly reduces the number of LLM calls to ~11% of those required by state-of-the-art methods and achieves a substantial performance improvement of 12.24%. We open-source our code and data (https://github.com/kiranpurohit/EXPLORA).

EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoning

TL;DR

ExPLORA is introduced, a novel exploration method designed to estimate the parameters of the scoring function, which evaluates exemplar subsets without incorporating confidence information, and significantly reduces the number of LLM calls to ~11% of those required by state-of-the-art methods and achieves a substantial performance improvement of 12.24%.

Abstract

Answering reasoning-based complex questions over text and hybrid sources, including tables, is a challenging task. Recent advances in large language models (LLMs) have enabled in-context learning (ICL), allowing LLMs to acquire proficiency in a specific task using only a few demonstration samples (exemplars). A critical challenge in ICL is the selection of optimal exemplars, which can be either task-specific (static) or test-example-specific (dynamic). Static exemplars provide faster inference times and increased robustness across a distribution of test examples. In this paper, we propose an algorithm for static exemplar subset selection for complex reasoning tasks. We introduce EXPLORA, a novel exploration method designed to estimate the parameters of the scoring function, which evaluates exemplar subsets without incorporating confidence information. EXPLORA significantly reduces the number of LLM calls to ~11% of those required by state-of-the-art methods and achieves a substantial performance improvement of 12.24%. We open-source our code and data (https://github.com/kiranpurohit/EXPLORA).

Paper Structure

This paper contains 35 sections, 6 equations, 8 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. explora: Model-based Exploration for Exemplar Subset Selection
  4. ICL for Complex Reasoning
  5. Loss model for top-$l$ Exemplar Subset Selection
  6. A Sampling-based bandit algorithm for Top-$l$ Exemplar Subset Selection
  7. Experimental Setup
  8. Experimental setting
  9. Baselines
  10. Results
  11. Performance Comparison
  12. Transferability of exemplars from smaller LLMs to larger LLMs
  13. Robustness of exemplars selected by explora compared to other approaches
  14. Exemplar selection efficiency based on number of LLM evaluations
  15. Ablation Studies
  16. ...and 20 more sections

Figures (8)

  • Figure 1: Overview of explora: Initially, set $U$ is randomly selected from set $\mathcal{U}$. In each iteration, parameters of the scoring function $\sigma(\alpha,.)$ are computed by minimizing a loss function. $\sigma$ guides the selection of the subset from $\mathcal{U}\setminus U$ with the lowest loss, which is then used to update $U$. This iterative updating process ensures $U$ maintains low-loss subsets, leading to a more accurate estimation of $\alpha$ in subsequent iterations.
  • Figure 2: (Top) Frugal exemplar selection by explora: LLM calls LENS vs explora (y-axis) with corresponding EM scores indicated on top of bars. (Bottom) Runtime comparison LENS vs explora.
  • Figure 3: Plot showing number of LLM calls vs Exact Match for TabMWP and GSM8K
  • Figure 4: Prompt for AquaRat
  • Figure 5: Prompt for FinQA
  • ...and 3 more figures

Theorems & Definitions (1)

  • Definition 3.1: ICCR