Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?

Weizheng Gu, Chengze Li, Zhuohao Yu, Mengyuan Sun, Zhibang Yang, Wei Wang, Hongrui Jia, Shikun Zhang, Wei Ye

TL;DR

The paper tackles the ambiguity in evaluating trajectory-based tuning for LLM agents by showing that improved benchmark scores can arise from semantic learning or from exploiting training-time interface shortcuts. It introduces PIPE, a minimal, semantics-preserving interface perturbation protocol, and Interface Reliance (IR), a geometric-mean metric, to diagnose the extent to which agents rely on interface surface forms. Across 16 environments from AgentBench and AgentGym, PIPE reveals that trajectory-SFT often induces brittle interface shortcutting, with performance gaps widening under perturbations. The work demonstrates that combining PIPE and IR yields a more faithful assessment of true semantic tool-use, guiding training practices toward robust, interface-agnostic capabilities and stressing the need for interface-aware evaluation in agent benchmarks.

Abstract

Large language models are increasingly evaluated as interactive agents, yet standard agent benchmarks conflate two qualitatively distinct sources of success: semantic tool-use and interface-specific interaction pattern memorization. Because both mechanisms can yield identical task success on the original interface, benchmark scores alone are not identifiable evidence of environment-invariant capability. We propose PIPE, a protocol-level evaluation augmentation for diagnosing interface reliance by minimally rewriting environment interfaces while preserving task semantics and execution behavior. Across 16 environments from AgentBench and AgentGym and a range of open-source and API-based agents, PIPE reveals that trajectory-SFT substantially amplifies interface shortcutting: trained agents degrade sharply under minimal interface rewrites, while non-trajectory-trained models remain largely stable. We further introduce Interface Reliance (IR), a counterbalanced alias-based metric that quantifies preference for training-time interfaces, and show that interface shortcutting exhibits environment-dependent, non-monotonic training dynamics that remain invisible under standard evaluation. Our code is available at https://anonymous.4open.science/r/What-Do-Agents-Learn-from-Trajectory-SFT-Semantics-or-Interfaces--0831/.

What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?

TL;DR

The paper tackles the ambiguity in evaluating trajectory-based tuning for LLM agents by showing that improved benchmark scores can arise from semantic learning or from exploiting training-time interface shortcuts. It introduces PIPE, a minimal, semantics-preserving interface perturbation protocol, and Interface Reliance (IR), a geometric-mean metric, to diagnose the extent to which agents rely on interface surface forms. Across 16 environments from AgentBench and AgentGym, PIPE reveals that trajectory-SFT often induces brittle interface shortcutting, with performance gaps widening under perturbations. The work demonstrates that combining PIPE and IR yields a more faithful assessment of true semantic tool-use, guiding training practices toward robust, interface-agnostic capabilities and stressing the need for interface-aware evaluation in agent benchmarks.

Abstract

Large language models are increasingly evaluated as interactive agents, yet standard agent benchmarks conflate two qualitatively distinct sources of success: semantic tool-use and interface-specific interaction pattern memorization. Because both mechanisms can yield identical task success on the original interface, benchmark scores alone are not identifiable evidence of environment-invariant capability. We propose PIPE, a protocol-level evaluation augmentation for diagnosing interface reliance by minimally rewriting environment interfaces while preserving task semantics and execution behavior. Across 16 environments from AgentBench and AgentGym and a range of open-source and API-based agents, PIPE reveals that trajectory-SFT substantially amplifies interface shortcutting: trained agents degrade sharply under minimal interface rewrites, while non-trajectory-trained models remain largely stable. We further introduce Interface Reliance (IR), a counterbalanced alias-based metric that quantifies preference for training-time interfaces, and show that interface shortcutting exhibits environment-dependent, non-monotonic training dynamics that remain invisible under standard evaluation. Our code is available at https://anonymous.4open.science/r/What-Do-Agents-Learn-from-Trajectory-SFT-Semantics-or-Interfaces--0831/.
Paper Structure (61 sections, 8 equations, 5 figures, 22 tables)

This paper contains 61 sections, 8 equations, 5 figures, 22 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-based Agents and Trajectory-based Tuning
  4. Evaluation of Agents
  5. Perturb Interface Protocol for Evaluation
  6. Formulation of Agent Evaluation Protocol
  7. Minimal Interface Perturbation
  8. Distinguish Learning Effects
  9. Re-Evaluation on Agents
  10. Details of Re-Evaluation
  11. Results
  12. PIPE does not uniformly increase task difficulty.
  13. Trajectory-SFT agents are disproportionately brittle under interface perturbations.
  14. Failure Analysis under Interface Perturbations
  15. Quantifying Interface Reliance
  16. ...and 46 more sections

Figures (5)

  • Figure 1: Identical scores do not imply identical agent capabilities.
  • Figure 2: How does PIPE distinguish semantic learning and interface shortcutting.
  • Figure 3: Average number of "legacy" action calls per task under interface perturbations on three representative environments.
  • Figure 4: Qwen3-8b's performance and IR on DataBase and OperatingSystem across training epochs.
  • Figure 5: Epoch-wise performance and interface reliance on SciWorld for Qwen3-8B and Gemma3-4B. Top-left: success ratio of Qwen3-8B under Origin, Perturb 1, and Perturb 2. Top-right: IR of Qwen3-8B under different $\alpha$. Bottom-left: success ratio of Gemma3-4B under the same three interfaces. Bottom-right: IR of Gemma3-4B under different $\alpha$.