Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Small Agent Collaboration Beat a Single Big LLM?

Agata Żywot, Xinyi Chen, Maarten de Rijke

TL;DR

The paper tackles whether small, tool-augmented agents can rival a single large LLM on GAIA. It combines Qwen3 backbones with a tool-augmented Agentic-Reasoning framework to dissect the impact of model size, explicit thinking, and tool usage, showing that tool use yields the most reliable gains and can let 4B models outperform larger peers without tools. Explicit thinking provides nuanced benefits dependent on task difficulty and agent role, often hindering performance when misaligned with tool coordination. The findings advocate for selective, policy-driven thinking in agentic systems and emphasize robust tool orchestration as the key driver of practical, scalable performance. The work informs design principles for efficient, tool-enabled AI systems beyond mere model scaling, while acknowledging GAIA-specific limits and the need for broader benchmarks.

Abstract

This report studies whether small, tool-augmented agents can match or outperform larger monolithic models on the GAIA benchmark. Using Qwen3 models (4B-32B) within an adapted Agentic-Reasoning framework, we isolate the effects of model scale, explicit thinking (no thinking, planner-only, or full), and tool use (search, code, mind-map). Tool augmentation provides the largest and most consistent gains. Using tools, 4B models can outperform 32B models without tool access on GAIA in our experimental setup. In contrast, explicit thinking is highly configuration- and difficulty-dependent: planner-only thinking can improve decomposition and constraint tracking, while unrestricted full thinking often degrades performance by destabilizing tool orchestration, leading to skipped verification steps, excessive tool calls, non-termination, and output-format drift.

Can Small Agent Collaboration Beat a Single Big LLM?

TL;DR

The paper tackles whether small, tool-augmented agents can rival a single large LLM on GAIA. It combines Qwen3 backbones with a tool-augmented Agentic-Reasoning framework to dissect the impact of model size, explicit thinking, and tool usage, showing that tool use yields the most reliable gains and can let 4B models outperform larger peers without tools. Explicit thinking provides nuanced benefits dependent on task difficulty and agent role, often hindering performance when misaligned with tool coordination. The findings advocate for selective, policy-driven thinking in agentic systems and emphasize robust tool orchestration as the key driver of practical, scalable performance. The work informs design principles for efficient, tool-enabled AI systems beyond mere model scaling, while acknowledging GAIA-specific limits and the need for broader benchmarks.

Abstract

This report studies whether small, tool-augmented agents can match or outperform larger monolithic models on the GAIA benchmark. Using Qwen3 models (4B-32B) within an adapted Agentic-Reasoning framework, we isolate the effects of model scale, explicit thinking (no thinking, planner-only, or full), and tool use (search, code, mind-map). Tool augmentation provides the largest and most consistent gains. Using tools, 4B models can outperform 32B models without tool access on GAIA in our experimental setup. In contrast, explicit thinking is highly configuration- and difficulty-dependent: planner-only thinking can improve decomposition and constraint tracking, while unrestricted full thinking often degrades performance by destabilizing tool orchestration, leading to skipped verification steps, excessive tool calls, non-termination, and output-format drift.
Paper Structure (25 sections, 3 figures, 1 table)

This paper contains 25 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tool-Augmented Language Models
  4. Explicit Thinking and Planning
  5. Multi-Agent Collaboration
  6. Methodology
  7. Framework Adaptation
  8. Tool-Agent Integration
  9. Experiments
  10. Experimental Setup
  11. Models.
  12. Datasets.
  13. Evaluation Metrics.
  14. Reasoning and Tool Usage.
  15. Experimental Results
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Tool usage distribution across reasoning configurations. Larger models (14B, 32B) maintain stable ratios, mid-sized model (8B) increases tool diversity under explicit reasoning, and smaller models (4B) remain mostly search-focused. Total tool calls increase with explicit thinking.
  • Figure 2: Tool usage per GAIA difficulty level. Search dominates at all levels, coding increases for Levels 2–3, and Mind-Map usage rises for the 8B model under explicit reasoning. Larger models show more stable tool-use patterns across levels.
  • Figure 3: Average accuracy vs. total tool calls per difficulty level. Tool usage peaks at Level 2, while Level 3 accuracy drops despite similar or lower tool calls, suggesting early planning errors rather than execution issues.