Offline Training of Language Model Agents with Functions as Learnable Weights

TL;DR

This work introduces a paradigm to train LLM-based agents by learning and progressively updating a set of functional tools rather than altering LLM weights. An AgentOptimizer, guided by the agent’s execution history, performs add/revise/remove actions to shape these functions, with roll-back and early-stop mechanisms to maintain performance. Across three tasks (MATH, GAIA, TabMWP) and two agent systems (GPT-4+ and ReAct), the method yields noticeable improvements and demonstrates generalizability, transferability, and robustness compared with tool-creation baselines. The analysis shows learned functions become more employed and maintain low complexity, underscoring the practicality of function-weight-like optimization in zero-shot or black-box LLM settings.

Abstract

Researchers and practitioners have recently reframed powerful Large Language Models (LLMs) as agents, enabling them to automate complex tasks largely via the use of specialized functions. To facilitate the development of LLM agents, we present a novel paradigm of training LLM agents without modifying the LLM weights, which is particularly useful when the LLMs are difficult or inaccessible for modifications. Inspired by how humans continuously forge tools to adapt to real-world tasks, rather than change our biological structure to fit a static set of tools, we propose to progressively forge agent's functions to better solve the downstream tasks instead of modifying the LLM weights. By treating the functions as learnable `agent parameters' and leveraging the fundamental idea of model training in artificial intelligence, we develop AgentOptimizer that employs the LLM to update agents' functions and devise an agent training algorithm with two strategies, roll-back, and early-stop, to streamline the training process. With extensive experiments, we showcase that the agent training paradigm could significantly improve the performance of representative LLM agents in various downstream tasks. We also study the behavior of the agent training regarding aspects like the learning curve and domain transferability.

Figures (8)

• Figure 1: The comparison between model training and agent training. In model training, numerical optimizersruder2016overview such as SGD and Adam optimize the model weights according to the loss on the training set. In contrast, agent training iteratively updates the agents' functions according to the execution history using the proposed AgentOptimizer.
• Figure 2: On the MATH dataset, we visualize the changes in train/test performance across epochs when training a GPT-4+ agent. For analysis purposes, we select one data type where the training does improve the test performance (Positive) and another that does not (Negative).
• Figure 3: To investigate the domain transferability of the agent training method, we show test performances of three different data types of the MATH dataset after training with different domains.
• Figure 4: The comparisons between the "regular training" of our method and the extended "batch training". The batch training with an enlarged training set doesn't necessarily lead to better performance in different batch settings.
• Figure 5: After removing the roll-back and early-exit mechanisms, the learning curve of the training performance and the final test performance of GPT-4+ Agents.

Theorems & Definitions (4)

• Lemma 1.3
• proof : Proof of Lemma \ref{['lemma:difference_between_empirical_and_expected']}
• Theorem 1.4
• proof : Proof of Theorem \ref{['theorem:diff_expfunc_optimalfunc']}