Multidimensional Consistency Improves Reasoning in Language Models

TL;DR

This paper tackles the brittleness of math reasoning in LLMs by introducing Multidimensional Reasoning Consistency (MRC), a framework that systematically varies prompts along three dimensions—order of exemplars (COC), problem rephrasing (CPC), and language (CLC)—to induce diverse reasoning paths. It formalizes reasoning consistency (RC) and demonstrates that aggregating consistency across these dimensions yields improvements in accuracy on GSM8K (monolingual) and MGSM (multilingual) benchmarks, with larger gains for smaller models. Empirical results show COC generally yields the strongest RC, CPC offers substantial accuracy gains, and CLC provides multilingual benefits with language-induced variability, while combining all three (MRC) often produces the largest improvements. The work highlights that diversity of reasoning paths can be leveraged to enhance mathematical reasoning and suggests future directions for measuring path diversity and selectively integrating dimensions for even better robustness.

Abstract

While Large language models (LLMs) have proved able to address some complex reasoning tasks, we also know that they are highly sensitive to input variation, which can lead to different solution paths and final answers. Answer consistency across input variations can thus be taken as a sign of stronger confidence. Leveraging this insight, we introduce a framework, {\em Multidimensional Reasoning Consistency} where, focusing on math problems, models are systematically pushed to diversify solution paths towards a final answer, thereby testing them for answer consistency across multiple input variations. We induce variations in (i) order of shots in prompt, (ii) problem phrasing, and (iii) languages used. Extensive experiments on a large range of open-source state-of-the-art LLMs of various sizes show that reasoning consistency differs by variation dimension, and that by aggregating consistency across dimensions, our framework consistently enhances mathematical reasoning performance on both monolingual dataset GSM8K and multilingual dataset MGSM, especially for smaller models.

Figures (10)

• Figure 1: Example of variations: A math problem is presented in different forms or languages, resulting in different reasoning paths to solve it.
• Figure 2: Overview of our Multidimensional Reasoning Consistency (MRC) framework: (i) COC changes the exemplars order; (ii) CPC rewrites the given questions in the same language; and (iii) CLC rewrites the given questions in different languages.
• Figure 3: Reasoning consistency on three dimensions of variation. Note that COC and CPC are evaluated on the monolingual benchmark GSM8K, while CLC is evaluated on the multilingual benchmark MGSM.
• Figure 4: Reasoning accuracy of 4-shot for 8 different exemplars orders on GSM8K: (i) minimum score (MIN), (ii) mean score (MEAN), (iii) maximum score (MAX), and (iv) cross-order consistency (COC).
• Figure 5: Reasoning accuracy of using varying numbers of reasoning paths.