Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration

Yifei Chen, Guanting Dong, Zhicheng Dou

TL;DR

ET-Agent addresses misalignment in Tool-Integrated Reasoning by calibrating agent behaviors through a self-evolving data flywheel and a two-phase training framework. The approach expands exploration of tool-use trajectories and iteratively refines behaviors via Pareto sampling and curriculum RL. Experiments across six challenging tasks show improvements in correctness, efficiency, reasoning conciseness, and tool execution accuracy, demonstrating the viability of behavior calibration for TIR. The work provides a practical blueprint for TIR research and open-source code.

Abstract

Large Language Models (LLMs) can extend their parameter knowledge limits by adopting the Tool-Integrated Reasoning (TIR) paradigm. However, existing LLM-based agent training framework often focuses on answers' accuracy, overlooking specific alignment for behavior patterns. Consequently, agent often exhibits ineffective actions during TIR tasks, such as redundant and insufficient tool calls. How to calibrate erroneous behavioral patterns when executing TIR tasks, thereby exploring effective trajectories, remains an open-ended problem. In this paper, we propose ET-Agent, a training framework for calibrating agent's tool-use behavior through two synergistic perspectives: Self-evolving Data Flywheel and Behavior Calibration Training. Specifically, we introduce a self-evolutionary data flywheel to generate enhanced data, used to fine-tune LLM to improve its exploration ability. Based on this, we implement an two-phases behavior-calibration training framework. It is designed to progressively calibrate erroneous behavioral patterns to optimal behaviors. Further in-depth experiments confirm the superiority of \ourmodel{} across multiple dimensions, including correctness, efficiency, reasoning conciseness, and tool execution accuracy. Our ET-Agent framework provides practical insights for research in the TIR field. Codes can be found in https://github.com/asilverlight/ET-Agent

ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration

TL;DR

ET-Agent addresses misalignment in Tool-Integrated Reasoning by calibrating agent behaviors through a self-evolving data flywheel and a two-phase training framework. The approach expands exploration of tool-use trajectories and iteratively refines behaviors via Pareto sampling and curriculum RL. Experiments across six challenging tasks show improvements in correctness, efficiency, reasoning conciseness, and tool execution accuracy, demonstrating the viability of behavior calibration for TIR. The work provides a practical blueprint for TIR research and open-source code.

Abstract

Large Language Models (LLMs) can extend their parameter knowledge limits by adopting the Tool-Integrated Reasoning (TIR) paradigm. However, existing LLM-based agent training framework often focuses on answers' accuracy, overlooking specific alignment for behavior patterns. Consequently, agent often exhibits ineffective actions during TIR tasks, such as redundant and insufficient tool calls. How to calibrate erroneous behavioral patterns when executing TIR tasks, thereby exploring effective trajectories, remains an open-ended problem. In this paper, we propose ET-Agent, a training framework for calibrating agent's tool-use behavior through two synergistic perspectives: Self-evolving Data Flywheel and Behavior Calibration Training. Specifically, we introduce a self-evolutionary data flywheel to generate enhanced data, used to fine-tune LLM to improve its exploration ability. Based on this, we implement an two-phases behavior-calibration training framework. It is designed to progressively calibrate erroneous behavioral patterns to optimal behaviors. Further in-depth experiments confirm the superiority of \ourmodel{} across multiple dimensions, including correctness, efficiency, reasoning conciseness, and tool execution accuracy. Our ET-Agent framework provides practical insights for research in the TIR field. Codes can be found in https://github.com/asilverlight/ET-Agent
Paper Structure (85 sections, 5 equations, 9 figures, 9 tables, 3 algorithms)

This paper contains 85 sections, 5 equations, 9 figures, 9 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tool-Integrated Reasoning.
  4. Agentic Reinforcement Learning.
  5. Preliminary Experiment
  6. Analysis of Improper Tool-Use.
  7. Analysis of Flawed Reasoning Logic.
  8. Analysis of Trajectories' Diversity.
  9. The Proposed Method: ET-Agent
  10. Self-Evolving Data Flywheel
  11. Initialization.
  12. Correct Reasoning Enhancement.
  13. Incorrect Reasoning Reflection.
  14. Iterative Data Evolution.
  15. Behavior Calibration Training Framework
  16. ...and 70 more sections

Figures (9)

  • Figure 1: Illustration of two different error tool-use behavior patterns.
  • Figure 2: Statistics about redundant tool calls.
  • Figure 3: Distribution of additional tool calls required to modify incorrect outputs, and differences in tool call across trajectories for identical questions.
  • Figure 4: Statistics on Aborted Tool Execution Cases.
  • Figure 5: The overall process of ET-Agent. The left part represents Self-Evolving Data Flywheel, while the right part represents Behavior Calibration Training Framework.
  • ...and 4 more figures