Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks

TL;DR

DIPPER is introduced, a novel, training-free framework that transforms a single LLM into an effective inference-time ensemble, by feeding the model an optimized and diverse set of prompts in parallel, leading to performance gains.

Abstract

Large Language Models (LLMs), particularly smaller variants, still struggle with complex reasoning tasks. While inference-time prompting can guide reasoning, existing methods often rely on sequential queries. Ensemble approaches offer a promising path to performance gains, especially given recent batch inference speed-ups. This work introduces DIPPER, a novel, training-free framework that transforms a single LLM into an effective inference-time ensemble. By feeding the model an optimized and diverse set of prompts in parallel, DIPPER elicits varied reasoning paths, leading to performance gains. We empirically demonstrate significant improvements on reasoning benchmarks, such as MATH, where a DIPPER ensemble of three Qwen2-MATH-1.5B instances (via parallel prompting of a single model) outperforms a larger 7B model.

Figures (14)

• Figure 1: The accuracy distribution of 200 candidate prompts on MATH with Qwen2-MATH-1.5B.
• Figure 2: Comparison of different ensembles of 7 reasoning prompts on MATH.
• Figure 3: Accuracy vs. average number of unique answers using different numbers of prompts in ensembles.
• Figure 4: Comparison of different ensemble methods on MATH for Qwen2-MATH-1.5B.
• Figure 5: Comparison of different ensemble methods on MATH for the LLaMA3.2B model.