Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assertion-Conditioned Compliance: A Provenance-Aware Vulnerability in Multi-Turn Tool-Calling Agents

Daud Waqas, Aaryamaan Golthi, Erika Hayashida, Huanzhi Mao

TL;DR

The paper formalizes Assertion-Conditioned Compliance (A-CC) to diagnose how multi-turn tool-calling agents treat misleading cues from two sources: user-originated assertions and internal tool/system hints. Using the BFCL benchmark, it introduces two parallel assertion streams (USA and FSA) and a compliance-rate metric to separate procedural obedience from task success. The results reveal consistent, provenance-sensitive vulnerabilities across model families, with notable degradation in final state despite high nominal accuracy, underscoring a latent safety risk in production deployments. The work argues for provenance-aware guardrails and evaluation strategies to mitigate unnecessary or unsafe tool executions in real-world pipelines.

Abstract

Multi-turn tool-calling LLMs (models capable of invoking external APIs or tools across several user turns) have emerged as a key feature in modern AI assistants, enabling extended dialogues from benign tasks to critical business, medical, and financial operations. Yet implementing multi-turn pipelines remains difficult for many safety-critical industries due to ongoing concerns regarding model resilience. While standardized benchmarks such as the Berkeley Function-Calling Leaderboard (BFCL) have underpinned confidence concerning advanced function-calling models (like Salesforce's xLAM V2), there is still a lack of visibility into multi-turn conversation-level robustness, especially given their exposure to real-world systems. In this paper, we introduce Assertion-Conditioned Compliance (A-CC), a novel evaluation paradigm for multi-turn function-calling dialogues. A-CC provides holistic metrics that evaluate a model's behavior when confronted with misleading assertions originating from two distinct vectors: (1) user-sourced assertions (USAs), which measure sycophancy toward plausible but misinformed user beliefs, and (2) function-sourced assertions (FSAs), which measure compliance with plausible but contradictory system policies (e.g., stale hints from unmaintained tools). Our results show that models are highly vulnerable to both USA sycophancy and FSA policy conflicts, confirming A-CC as a critical, latent vulnerability in deployed agents.

Assertion-Conditioned Compliance: A Provenance-Aware Vulnerability in Multi-Turn Tool-Calling Agents

TL;DR

The paper formalizes Assertion-Conditioned Compliance (A-CC) to diagnose how multi-turn tool-calling agents treat misleading cues from two sources: user-originated assertions and internal tool/system hints. Using the BFCL benchmark, it introduces two parallel assertion streams (USA and FSA) and a compliance-rate metric to separate procedural obedience from task success. The results reveal consistent, provenance-sensitive vulnerabilities across model families, with notable degradation in final state despite high nominal accuracy, underscoring a latent safety risk in production deployments. The work argues for provenance-aware guardrails and evaluation strategies to mitigate unnecessary or unsafe tool executions in real-world pipelines.

Abstract

Multi-turn tool-calling LLMs (models capable of invoking external APIs or tools across several user turns) have emerged as a key feature in modern AI assistants, enabling extended dialogues from benign tasks to critical business, medical, and financial operations. Yet implementing multi-turn pipelines remains difficult for many safety-critical industries due to ongoing concerns regarding model resilience. While standardized benchmarks such as the Berkeley Function-Calling Leaderboard (BFCL) have underpinned confidence concerning advanced function-calling models (like Salesforce's xLAM V2), there is still a lack of visibility into multi-turn conversation-level robustness, especially given their exposure to real-world systems. In this paper, we introduce Assertion-Conditioned Compliance (A-CC), a novel evaluation paradigm for multi-turn function-calling dialogues. A-CC provides holistic metrics that evaluate a model's behavior when confronted with misleading assertions originating from two distinct vectors: (1) user-sourced assertions (USAs), which measure sycophancy toward plausible but misinformed user beliefs, and (2) function-sourced assertions (FSAs), which measure compliance with plausible but contradictory system policies (e.g., stale hints from unmaintained tools). Our results show that models are highly vulnerable to both USA sycophancy and FSA policy conflicts, confirming A-CC as a critical, latent vulnerability in deployed agents.

Paper Structure

This paper contains 18 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Benchmark & Task Setup (BFCL)
  5. Assertion Generation
  6. User-Sourced Assertions (USAs)
  7. Function-Sourced Assertions (FSAs)
  8. Data Injection & Execution Protocol
  9. Metrics
  10. Results
  11. Conclusion
  12. Experimental Setup and Models
  13. BFCL Task Setup
  14. Execution Procedure
  15. Assertion Generation Pipeline
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Overview of baseline, user-sourced assertion (USA), and function-sourced assertion (FSA) behavior in multi-turn tool-calling. Assertions may cause compliance in the immediate function call, after which the model may either propagate the misleading path (degradation) or recover and produce the correct final environment state.
  • Figure 2: Run-to-run standard deviation ($\sigma$) of accuracy deltas across the $N=3$ independent BFCL executions for each model and assertion condition. Lower $\sigma$ indicates higher stability. Most models exhibit $\sigma$ < 2 pp across all conditions, confirming that assertion-induced degradations are systematic rather than stochastic. Qwen3 variants show moderately higher $\sigma$ values (up to 7 pp) in write-heavy and FSA settings, reflecting greater sensitivity to interaction-level variability.