Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments

TL;DR

The paper tackles the challenge of enabling robust tool use in large language models by introducing an automated five-stage environment-construction pipeline and a verifiable reward mechanism, enabling stable, feedback-driven training without relying on external tools. It couples this environment with trajectory-based preference optimization to improve tool-use performance across model scales and training algorithms, with gains largely attributed to updates in lower-layer MLP parameters that enhance contextual understanding. The approach maintains general capabilities and demonstrates strong out-of-domain generalization, though reasoning-mode benefits for tool use are nuanced and task-dependent. Overall, the work presents a scalable, self-contained framework that strengthens LLMs' tool-use abilities while providing insights into how early-layer representations drive these improvements.

Abstract

Effective tool use is essential for large language models (LLMs) to interact meaningfully with their environment. However, progress is limited by the lack of efficient reinforcement learning (RL) frameworks specifically designed for tool use, due to challenges in constructing stable training environments and designing verifiable reward mechanisms. To address this, we propose an automated environment construction pipeline, incorporating scenario decomposition, document generation, function integration, complexity scaling, and localized deployment. This enables the creation of high-quality training environments that provide detailed and measurable feedback without relying on external tools. Additionally, we introduce a verifiable reward mechanism that evaluates both the precision of tool use and the completeness of task execution. When combined with trajectory data collected from the constructed environments, this mechanism integrates seamlessly with standard RL algorithms to facilitate feedback-driven model training. Experiments on LLMs of varying scales demonstrate that our approach significantly enhances the models' tool-use performance without degrading their general capabilities, regardless of inference modes or training algorithms. Our analysis suggests that these gains result from improved context understanding and reasoning, driven by updates to the lower-layer MLP parameters in models.

Figures (5)

• Figure 1: Illustrative examples of four scenarios, categorized by varying sub-question pattern combinations.
• Figure 2: Overview of our proposed approach. The automated environment construction follows a five-stage pipeline to generate diverse tool-use training environments. Feedback-driven model training then collects data within these environments, incorporates verifiable reward mechanisms, and optimizes performance using preference-based RL algorithms.
• Figure 3: Performance of each generalized capability before and after training across different models.
• Figure 4: Performance of Qwen 2.5-7B trained using different reward mechanisms.
• Figure 5: Solve-F1 of various LLMs across training epochs.