Quasi-random Multi-Sample Inference for Large Language Models

TL;DR

Quasi-random Multi-Sample Inference for Large Language Models introduces arithmetic sampling, which constructs a codebook over the unit interval $[0,1)$ to generate diverse, embarrassingly parallel samples for LLM decoding. It applies this mechanism to chain-of-thought self-consistency and minimum Bayes risk decoding for translation, comparing against ancestral sampling across GSM8K, CommonsenseQA, and WMT19 tasks. The results show that arithmetic sampling increases sample diversity and yields performance gains—3–5 percentage points on GSM8K and 0.45–0.89 COMET-point improvements in MT—without meaningful overhead, with larger benefits as sample count grows. Limitations include random vocabulary ordering and one-dimensional token spaces, suggesting future work on semantically informed orderings and higher-dimensional embeddings, as well as extensions to MAP decoding.

Abstract

Large language models (LLMs) are often equipped with multi-sample decoding strategies. An LLM implicitly defines an arithmetic code book, facilitating efficient and embarrassingly parallelizable \textbf{arithmetic sampling} to produce multiple samples using quasi-random codes. Traditional text generation methods, such as beam search and sampling-based techniques, have notable limitations: they lack parallelizability or diversity of sampled sequences. This study explores the potential of arithmetic sampling, contrasting it with ancestral sampling across two decoding tasks that employ multi-sample inference: chain-of-thought reasoning with self-consistency and machine translation with minimum Bayes risk decoding. Our results demonstrate that arithmetic sampling produces more diverse samples, significantly improving reasoning and translation performance as the sample size increases. We observe a $\mathbf{3\text{-}5\%}$ point increase in accuracy on the GSM8K dataset and a $\mathbf{0.45\text{-}0.89\%}$ point increment in COMET score for WMT19 tasks using arithmetic sampling without any significant computational overhead.

Figures (6)

• Figure 1: 8-shot evaluation on GSM8K with Gemma-7B and Llama-2-7B
• Figure 2: 6-shot evaluation on Commonsense QA with Gemma-7B and Llama-2-7B
• Figure 3: COMET vs. #sampled sequences on Flan T5, MT0
• Figure 4: COMET vs. n-gram diversity on Flan-T5, MT0 varying temperature $T$
• Figure 5: COMET vs. #sampled sequences on Flan-T5, MT0