Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Budget-Constrained Tool Learning with Planning

Yuanhang Zheng, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu

TL;DR

Budget-Constrained Tool Learning with Planning (BTP) addresses cost-aware tool learning for LLMs by preplanning tool usage before invocation. It introduces a planning phase that estimates tool usefulness $v(q_u,t_i)$ from past experience and a frequency cap $\tilde{F}(q_u,t_i)$, then solves a knapsack-like DP to maximize total value under remaining budget $R=B-c_s$. The approach integrates with existing tool-learning methods and is augmented with a blacklist mechanism to prune unhelpful tools, reducing cost further. Experiments on ToolBench show that BTP improves budgeted pass rate and reduces average cost across diverse baselines, demonstrating practical benefit under budget constraints.

Abstract

Despite intensive efforts devoted to tool learning, the problem of budget-constrained tool learning, which focuses on resolving user queries within a specific budget constraint, has been widely overlooked. This paper proposes a novel method for budget-constrained tool learning. Our approach involves creating a preferable plan under the budget constraint before utilizing the tools. This plan outlines the feasible tools and the maximum number of times they can be employed, offering a comprehensive overview of the tool learning process for large language models. This allows them to allocate the budget from a broader perspective. To devise the plan without incurring significant extra costs, we suggest initially estimating the usefulness of the candidate tools based on past experience. Subsequently, we employ dynamic programming to formulate the plan. Experimental results demonstrate that our method can be integrated with various tool learning methods, significantly enhancing their effectiveness under strict budget constraints.

Budget-Constrained Tool Learning with Planning

TL;DR

Budget-Constrained Tool Learning with Planning (BTP) addresses cost-aware tool learning for LLMs by preplanning tool usage before invocation. It introduces a planning phase that estimates tool usefulness from past experience and a frequency cap , then solves a knapsack-like DP to maximize total value under remaining budget . The approach integrates with existing tool-learning methods and is augmented with a blacklist mechanism to prune unhelpful tools, reducing cost further. Experiments on ToolBench show that BTP improves budgeted pass rate and reduces average cost across diverse baselines, demonstrating practical benefit under budget constraints.

Abstract

Despite intensive efforts devoted to tool learning, the problem of budget-constrained tool learning, which focuses on resolving user queries within a specific budget constraint, has been widely overlooked. This paper proposes a novel method for budget-constrained tool learning. Our approach involves creating a preferable plan under the budget constraint before utilizing the tools. This plan outlines the feasible tools and the maximum number of times they can be employed, offering a comprehensive overview of the tool learning process for large language models. This allows them to allocate the budget from a broader perspective. To devise the plan without incurring significant extra costs, we suggest initially estimating the usefulness of the candidate tools based on past experience. Subsequently, we employ dynamic programming to formulate the plan. Experimental results demonstrate that our method can be integrated with various tool learning methods, significantly enhancing their effectiveness under strict budget constraints.
Paper Structure (25 sections, 13 equations, 5 figures, 7 tables, 2 algorithms)

This paper contains 25 sections, 13 equations, 5 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Budget-Constrained Tool Learning
  4. Tool Usage Planning
  5. Applying Plan to Tool Learning
  6. Blacklist Mechanism
  7. Experiments
  8. Setup
  9. Data Preparation.
  10. Baselines.
  11. Evaluation.
  12. Implementation Details.
  13. Main Results
  14. Planned v.s. Actual Tool Usage
  15. Effect of Budget Constraint $R$
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Example of budget-constrained tool learning. In this example, the baseline fails to resolve the user query within the budget constraint. Our proposed BTP makes a preferable plan of tool usage before using the tools, which may help resolve the user query under the budget constraint. "Expected Value" measures how valuable the candidate tool is for resolving the given query, which is estimated based on the past experience of tool learning. The shaded area in the "Planning" part means that the candidate plans in the area exceed the budget constraint.
  • Figure 2: Effect of budget constraint $R$ on Pass rate under Budget Constraint (PBC) and Average Cost (AC). The results are evaluated on the I1-Inst subset.
  • Figure 3: Effect of threshold $\tau$ on Pass rate under Budget Constraint (PBC) and Average Cost (AC). The results are evaluated on the I1-Inst subset.
  • Figure 4: The extra content which may be appended to the system prompt during the tool learning process.
  • Figure 5: Case study on the I1-Inst subset.