Blue Teaming Function-Calling Agents

TL;DR

This study analyzes the security of open-source function-calling LLMs by evaluating three attack types against four models and eight defences. It provides a focused, tool-centric empirical assessment, revealing that default safety is lacking and that existing defences are not universally effective. The findings highlight the emergence of attack vectors like Renaming Tool Poisoning and demonstrate that approaches such as Description Rewriting and Watermarking can be promising, yet require further refinement. The work underscores the need for specialized datasets and tailored defence strategies to achieve robust, trustworthy agentic systems in real-world deployments.

Abstract

We present an experimental evaluation that assesses the robustness of four open source LLMs claiming function-calling capabilities against three different attacks, and we measure the effectiveness of eight different defences. Our results show how these models are not safe by default, and how the defences are not yet employable in real-world scenarios.