Pheromone-based Learning of Optimal Reasoning Paths

TL;DR

This work introduces ACO-ToT, a neuroscience-inspired framework that fuses Ant Colony Optimization with Tree of Thought to efficiently discover high-quality reasoning paths for complex problems in large language models. By deploying a colony of specialized LLMs (ants) to explore a ToT-derived reasoning graph and reinforce productive steps via pheromones, the method achieves substantial accuracy gains over CoT, ToT, and IRPO across GSM8K, ARC-Challenge, and MATH, while providing insights into convergence speed and path properties. Key contributions include a mixture-of-experts scoring mechanism for path evaluation, theoretical analysis of convergence and costs, and extensive experiments with ablations that highlight the importance of expert diversity and exploration-balanced parameters. The approach demonstrates that incorporating biologically inspired collective search into LLM inference can meaningfully enhance reasoning performance, with potential implications for automated theorem proving, education, and scientific reasoning systems, albeit with notable computational costs and design considerations for real-world deployment.

Abstract

Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities through chain-of-thought prompting, yet discovering effective reasoning methods for complex problems remains challenging due to the vast space of possible intermediate steps. We introduce Ant Colony Optimization-guided Tree of Thought (ACO-ToT), a novel algorithm that combines ACO with LLMs to discover optimal reasoning paths for complex problems efficiently. Drawing inspiration from Hebbian learning in neurological systems, our method employs a collection of distinctly fine-tuned LLM "ants" to traverse and lay pheromone trails through a centralized tree of thought, with each ant's movement governed by a weighted combination of existing pheromone trails and its own specialized expertise. The algorithm evaluates complete reasoning paths using a mixture-of-experts-based scoring function, with pheromones reinforcing productive reasoning paths across iterations. Experiments on three challenging reasoning tasks (GSM8K, ARC-Challenge, and MATH) demonstrate that ACO-ToT performs significantly better than existing chain-of-thought optimization approaches, suggesting that incorporating biologically inspired collective search mechanisms into LLM inference can substantially enhance reasoning capabilities.

Figures (3)

• Figure 1: Comparison of approaches to complex reasoning problems with LLMs. Each rectangular node represents a thought---an intermediate reasoning step to solve a larger problem. On the right, our method utilizes "ants" traversing between connected reasoning steps (depicted as circles) to strengthen productive reasoning steps across iterations via pheromone trails. See algorithmic implementation for \ref{['fig:acototoverview']} in Figure \ref{['fig:algflowchart']}.
• Figure 2: Example procedure for a math problem from GSM8K. The central LLM is prompted for an initial ToT, which is next explored by fine-tuned ant LLMs to discover an optimal reasoning path, and then computed for a final result. See Algorithm \ref{['alg:mainaco']} for general procedure.
• Figure 3: Convergence analysis of ACO-ToT showing (left) performance metrics and (right) path properties across iterations. The algorithm typically converges after 3-4 iterations, with accuracy improving from 55.6% to 81.6%, expert agreement reaching 86%, and coherence scores stabilizing at 82%. Path lengths decrease and stabilize at 4.4 steps on average, while the ratio of pheromone concentration between optimal and suboptimal paths reaches 2.4.