Scaling Trends for Multi-Hop Contextual Reasoning in Mid-Scale Language Models

TL;DR

This work investigates how multi-hop contextual reasoning scales in mid-scale language models using a synthetic, controlled framework that separates task structure from reasoning method. It contrasts rule-based pattern matching with multi-agent LLM reasoning across four model configurations, fitting both power-law and sigmoidal scaling models to characterize performance. Key findings include a robust task-method dissociation, where rule-based methods excel on structured tasks but fail on contextual tasks, while multi-agent LLMs excel in the latter; multi-agent gains depend on base capability and can be substantial (up to ~46.7 percentage points), with active parameters better predicting MoE reasoning than total parameters; architecture quality also matters beyond parameter count. The study provides a practical, reproducible evaluation framework runnable on consumer hardware and offers insights for deployment decisions, showing that advances in reasoning hinge on effective use of model capacity and that multi-agent coordination amplifies, rather than substitutes for, existing capabilities. Code and data are released at https://github.com/micahkatz/multi-hop-contextual-reasoning.git to support reproducible research.

Abstract

We present a controlled study of multi-hop contextual reasoning in large language models, providing a clean demonstration of the task-method dissociation: rule-based pattern matching achieves 100% success on structured information retrieval but only 6.7% on tasks requiring cross-document reasoning, while LLM-based multi-agent systems show the inverse pattern, achieving up to 80% on reasoning tasks where rule-based methods fail. Using a synthetic evaluation framework with 120 trials across four models (LLaMA-3 8B, LLaMA-2 13B, Mixtral 8x7B, DeepSeek-V2 16B), we report three key findings: (1) Multi-agent amplification depends on base capability: statistically significant gains occur only for models with sufficient reasoning ability (p < 0.001 for LLaMA-3 8B, p = 0.014 for Mixtral), with improvements of up to 46.7 percentage points, while weaker models show no benefit, suggesting amplification rather than compensation; (2) Active parameters predict reasoning performance: Mixtral's performance aligns with its ~12B active parameters rather than 47B total, consistent with the hypothesis that inference-time compute drives reasoning capability in MoE architectures; (3) Architecture quality matters: LLaMA-3 8B outperforms LLaMA-2 13B despite fewer parameters, consistent with known training improvements. Our results provide controlled quantitative evidence for intuitions about multi-agent coordination and MoE scaling, while highlighting the dependence of multi-agent benefits on base model capability. We release our evaluation framework to support reproducible research on reasoning in mid-scale models.

Figures (6)

• Figure 1: Comparison of structured (single-hop) and contextual (multi-hop) reasoning tasks. Structured tasks require only pattern matching on co-located information, while contextual tasks require linking disparate facts through implicit relationships.
• Figure 2: Multi-agent architecture for contextual reasoning. The Analyst extracts information, the Strategist generates hypotheses, and the Generator produces candidates. Failed attempts trigger iterative refinement through the feedback loop.
• Figure 3: Task-method dissociation. Left: On structured tasks, rule-based achieves 100% while LLM performance varies. Right: On contextual reasoning tasks, the pattern inverts, LLM multi-agent systems significantly outperform rule-based methods. Stars indicate statistical significance: *** $p < 0.001$, * $p < 0.05$.
• Figure 4: Performance prediction: total vs. active parameters. Left panel shows poor correlation between total parameters and reasoning success (Mixtral appears as an outlier). Right panel shows better alignment when plotting against active parameters, supporting the hypothesis that active parameter count drives reasoning capability.
• Figure 5: Performance vs reasoning complexity for LLaMA-3 8B. Multi-agent architecture maintains high success rates (60--100%) across 2--4 reasoning hops, while single-agent performance degrades from 80% at 2 hops to 0% at 4 hops. Rule-based methods achieve 100% at 1 hop (pattern matching) but fail completely at multi-hop tasks.