Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task-Aware LLM Council with Adaptive Decision Pathways for Decision Support

Wei Zhu, Lixing Yu, Hao-Ren Yao, Zhiwen Tang, Kun Yue

TL;DR

This work addresses the brittleness of applying a single LLM across diverse decision tasks by introducing TALC, a task-aware LLM council that couples expert routing with memory-guided planning. TALC maintains Success Memory Segments for each expert and uses retrieval-augmented routing to dynamically assign decision attributions, while a dual-signal value estimator combines model-based evaluations with memory priors to guide adaptive MCTS expansion. The approach yields higher task success rates and improved search efficiency across WebShop, HumanEval, and Game of 24, outperforming strong baselines in both lightweight and full configurations. By leveraging semantic matching of current reasoning contexts to past successes and adaptive planning depth, TALC delivers robust, efficient decision support with practical implications for complex, long-horizon tasks.

Abstract

Large language models (LLMs) have shown strong capabilities across diverse decision-making tasks. However, existing approaches often overlook the specialization differences among available models, treating all LLMs as uniformly applicable regardless of task characteristics. This limits their ability to adapt to varying reasoning demands and task complexities. In this work, we propose Task-Aware LLM Council (TALC), a task-adaptive decision framework that integrates a council of LLMs with Monte Carlo Tree Search (MCTS) to enable dynamic expert selection and efficient multi-step planning. Each LLM is equipped with a structured success memory profile derived from prior task trajectories, enabling semantic matching between current reasoning context and past successes. At each decision point, TALC routes control to the most contextually appropriate model and estimates node value using a dual-signal mechanism that fuses model-based evaluations with historical utility scores. These signals are adaptively weighted based on intra-node variance and used to guide MCTS selection, allowing the system to balance exploration depth with planning confidence. Experiments on WebShop, HumanEval, and the Game of 24 demonstrate that TALC achieves superior task success rates and improved search efficiency compared to strong baselines, validating the benefits of specialization-aware routing and adaptive planning.

Task-Aware LLM Council with Adaptive Decision Pathways for Decision Support

TL;DR

This work addresses the brittleness of applying a single LLM across diverse decision tasks by introducing TALC, a task-aware LLM council that couples expert routing with memory-guided planning. TALC maintains Success Memory Segments for each expert and uses retrieval-augmented routing to dynamically assign decision attributions, while a dual-signal value estimator combines model-based evaluations with memory priors to guide adaptive MCTS expansion. The approach yields higher task success rates and improved search efficiency across WebShop, HumanEval, and Game of 24, outperforming strong baselines in both lightweight and full configurations. By leveraging semantic matching of current reasoning contexts to past successes and adaptive planning depth, TALC delivers robust, efficient decision support with practical implications for complex, long-horizon tasks.

Abstract

Large language models (LLMs) have shown strong capabilities across diverse decision-making tasks. However, existing approaches often overlook the specialization differences among available models, treating all LLMs as uniformly applicable regardless of task characteristics. This limits their ability to adapt to varying reasoning demands and task complexities. In this work, we propose Task-Aware LLM Council (TALC), a task-adaptive decision framework that integrates a council of LLMs with Monte Carlo Tree Search (MCTS) to enable dynamic expert selection and efficient multi-step planning. Each LLM is equipped with a structured success memory profile derived from prior task trajectories, enabling semantic matching between current reasoning context and past successes. At each decision point, TALC routes control to the most contextually appropriate model and estimates node value using a dual-signal mechanism that fuses model-based evaluations with historical utility scores. These signals are adaptively weighted based on intra-node variance and used to guide MCTS selection, allowing the system to balance exploration depth with planning confidence. Experiments on WebShop, HumanEval, and the Game of 24 demonstrate that TALC achieves superior task success rates and improved search efficiency compared to strong baselines, validating the benefits of specialization-aware routing and adaptive planning.
Paper Structure (20 sections, 10 equations, 3 figures, 3 tables)

This paper contains 20 sections, 10 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM as a Single-Agent System
  4. Multi-Agent Collaboration
  5. Memory-Augmented LLMs
  6. Preliminaries
  7. Reinforcement Learning and LLM Agents
  8. Monte Carlo Tree Search (MCTS)
  9. Methodology
  10. Expert Profiling for LLM council
  11. Task-Aware Council Routing
  12. Dual-Signal Value Estimation
  13. Adaptive Search via Controlled Expansion
  14. Experiments
  15. Experiment Settings
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Overview of TALC.
  • Figure 2: Main Results. Metrics include success rate (Game of 24 and WebShop) and pass@1 (HumanEval). The WebShop reward score ($r \in [0,1]$) reflects partial fulfillment of user-specified attributes. Some baselines (e.g., MASTER) report only a subset of metrics due to unavailable code and are shown accordingly.
  • Figure 3: Efficiency Analysis. Baselines use K=10 and K=30 max iterations; our method uses only K=10.