Enhancing the General Agent Capabilities of Low-Parameter LLMs through Tuning and Multi-Branch Reasoning

TL;DR

This work targets closing the gap between open-source, low-parameter LLMs and commercial agents by combining supervised fine-tuning with agent-specific data and multi-path reasoning. Agent data is crafted using GPT-4 to train 7B/13B LLMs via LoRA in a two-dataset setup that blends agent-focused trajectories with general instructions, formalized by a loss $\mathcal{L}(\theta)= \lambda \cdot \mathbb{E}_{(x,y)\sim D_{agent}}[\log M_{\theta}(y|x)]+ (1-\lambda)\cdot \mathbb{E}_{(x,y)\sim D_{general}}[\log M_{\theta}(y|x)]$. To tackle complex tasks, the paper introduces task decomposition through a planning model $M_p$ that generates subtasks (with $k=3$) and a judge model $M_{jdg}$ that validates completion, augmented by backtracking to avoid non-optimal reasoning paths. Evaluations on AgentBench show that SFT with agent data reduces hallucinations and formatting errors, while multi-path reasoning further improves performance, particularly for 13B LLMs, indicating practical routes to deploying capable, open-source agents. The findings suggest that carefully constructed agent data and structured reasoning strategies can substantially elevate the real-world utility of low-parameter LLMs as autonomous agents.

Abstract

Open-source pre-trained Large Language Models (LLMs) exhibit strong language understanding and generation capabilities, making them highly successful in a variety of tasks. However, when used as agents for dealing with complex problems in the real world, their performance is far inferior to large commercial models such as ChatGPT and GPT-4. As intelligent agents, LLMs need to have the capabilities of task planning, long-term memory, and the ability to leverage external tools to achieve satisfactory performance. Various methods have been proposed to enhance the agent capabilities of LLMs. On the one hand, methods involve constructing agent-specific data and fine-tuning the models. On the other hand, some methods focus on designing prompts that effectively activate the reasoning abilities of the LLMs. We explore both strategies on the 7B and 13B models. We propose a comprehensive method for constructing agent-specific data using GPT-4. Through supervised fine-tuning with constructed data, we find that for these models with a relatively small number of parameters, supervised fine-tuning can significantly reduce hallucination outputs and formatting errors in agent tasks. Furthermore, techniques such as multi-path reasoning and task decomposition can effectively decrease problem complexity and enhance the performance of LLMs as agents. We evaluate our method on five agent tasks of AgentBench and achieve satisfactory results.

Figures (5)

• Figure 1: The agent performance of open-source LLMs and commercial LLMs. Agent Overall Score is the average accuracy of several agent tasks.
• Figure 2: The process of constructing agent data. For task planning and external tool usage capabilities, we use two strategies, respectively.
• Figure 3: The process of task decomposition. The planning model breaks the entire task into several small subtasks.
• Figure 4: The comparison of different reasoning methods. From the left to right are Input Output (IO), ToT and our method.
• Figure 5: Comparison of ReAct and our method in agent task reasoning. We show the action and observation in webshop and household tasks.