Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Calls

TL;DR

Gecko is introduced, a comprehensive environment that simulates tool responses using a combination of rules and LLMs that allows LLMs to refine their tool calls, forming a simple yet effective test-time scaling method named GATS.

Abstract

The ability to use tools is fundamental for large language model (LLM) agents. Given a task, existing systems use LLMs to plan and generate tool calls, which are executed by real-world tools to complete the task. However, tool calls are prone to errors because they are derived merely from LLM intrinsic capabilities. What is more, while it is useful to let LLMs iteratively refine the tool-call sequence using execution results from real tools, this process can be expensive and lead to unsafe results. To improve LLM tool calls and address issues caused by using real tools for refinement, we introduce Gecko, a comprehensive environment that simulates tool responses using a combination of rules and LLMs. Specifically, Gecko checks the validity of tool calls including input arguments and tool names, synthesizes reasonable responses that adhere to the output schema, and assesses whether all task objectives have been achieved. These three types of feedback provided by Gecko allow LLMs to refine their tool calls, forming a simple yet effective test-time scaling method named GATS. On BFCLv3 and $τ^2$-bench, GATS consistently improves the tool calling performance of various LLMs including GPT-4o, GPT-5, and Gemini-3.0-pro. We further discuss working mechanisms of our method and share future possibilities.

Figures (5)

• Figure 1: GATS uses Gecko’s feedback from simulated tool executions to iteratively refine tool calls, consistently improving the performance of different LLMs on BFCLv3 and $\tau^2$-bench.
• Figure 2: Overview of tool-call refinement with Gecko feedback, illustrated through a single dialogue. From left to right, we show three consecutive refinement attempts. [Left:] Tool call fails the argument validation check due to an incorrect filename format. [Middle:] Arguments have passed validation, but simulated execution reveals that the folder is created in a wrong directory, so the solution does not solve the task. [Right:] The planning LLM finally generates a correct solution to be safely executed in a real environment.
• Figure 3: Examples of feedback provided by Gecko through (a) syntactic argument validation, (b) semantic argument validation, and (c) synthetic responses, task state, and task feedback. The validation feedback (orange), synthetic response (yellow), and task state and feedback (blue) will then be fed to the planning LLM during test-time scaling.
• Figure 4: (a) Ablation study of Gecko on BFCL-Multi-Turn-Base. The full system (72.0%) is compared against variants with one component removed: argument validation (70.5%), task state estimation (68.0%), and task feedback (61.5%); (b) LLM replacement study on Gecko evaluated on BFCLv3 'Multi-turn base'. We use GPT-4o as planning LLM. For each component, the bar on the left is the original performance 72.0%. Under LLM replacement, e.g., replacing GPT-4.1-nano with GPT-4o for argument validation.
• Figure 5: Test-time scaling behaviours. We evaluate GATS on the BFCL-Non-Live-Simple, using GPT-4o as the planning LLM. Retry times is the maximum number of feedback-based refinement steps allowed in GATS (Section \ref{['sec:tts']}). We report (a) accuracy (%), (b) average latency (s) and (c) average cost ($) per user task versus maximum retry times.