Reevaluating Self-Consistency Scaling in Multi-Agent Systems
Chiyan Loo
TL;DR
Problem: Whether increasing sampled reasoning paths via self-consistency yields meaningful gains for current LLMs. Approach: Empirically evaluate Gemini 2.5 variants on HotpotQA and Math-500, comparing multi-agent self-consistency with a single CoT baseline while tracking accuracy, token cost, and latency. Findings: Accuracy improves with more agents up to a plateau, with diminishing returns largely due to path overlap; larger models show smoother, more reliable gains. Significance: Demonstrates that self-consistency remains useful but high-sample configurations are often cost-inefficient, guiding practical deployment of multi-agent reasoning in real systems.
Abstract
This study examines the trade-offs of increasing sampled reasoning paths in self-consistency for modern large language models (LLMs). Earlier research with older models showed that combining multiple reasoning chains improves results before reaching a plateau. Using Gemini 2.5 models on HotpotQA and Math-500, we revisit those claims under current model conditions. Each configuration pooled outputs from varying sampled reasoning paths and compared them to a single chain-of-thought (CoT) baseline. Larger models exhibited a more stable and consistent improvement curve. The results confirm that performance gains taper off after moderate sampling, aligning with past findings. This plateau suggests diminishing returns driven by overlap among reasoning paths. Self-consistency remains useful, but high-sample configurations offer little benefit relative to their computational cost.