Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models

Zihui Wu, Haichang Gao, Jianping He, Ping Wang

TL;DR

The paper addresses the overlooked security risk of jailbreaking through LLM function calling by designing a Jailbreak Function attack and empirically evaluating it across six state-of-the-art models. It identifies three root causes—alignment discrepancies in function arguments, user coercion to execute functions, and insufficient safety filtering—and demonstrates high attack success rates, challenging assumptions about function-call safety. The authors propose defensive strategies, notably defensive prompts, and provide practical guidance for hardening function-calling workflows. Overall, the work highlights the need for security measures that cover all interaction modes with LLMs, including tool use, to prevent emergent jailbreak vectors in real-world deployments.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities, but their power comes with significant security considerations. While extensive research has been conducted on the safety of LLMs in chat mode, the security implications of their function calling feature have been largely overlooked. This paper uncovers a critical vulnerability in the function calling process of LLMs, introducing a novel "jailbreak function" attack method that exploits alignment discrepancies, user coercion, and the absence of rigorous safety filters. Our empirical study, conducted on six state-of-the-art LLMs including GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-pro, reveals an alarming average success rate of over 90\% for this attack. We provide a comprehensive analysis of why function calls are susceptible to such attacks and propose defensive strategies, including the use of defensive prompts. Our findings highlight the urgent need for enhanced security measures in the function calling capabilities of LLMs, contributing to the field of AI safety by identifying a previously unexplored risk, designing an effective attack method, and suggesting practical defensive measures. Our code is available at https://github.com/wooozihui/jailbreakfunction.

The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models

TL;DR

The paper addresses the overlooked security risk of jailbreaking through LLM function calling by designing a Jailbreak Function attack and empirically evaluating it across six state-of-the-art models. It identifies three root causes—alignment discrepancies in function arguments, user coercion to execute functions, and insufficient safety filtering—and demonstrates high attack success rates, challenging assumptions about function-call safety. The authors propose defensive strategies, notably defensive prompts, and provide practical guidance for hardening function-calling workflows. Overall, the work highlights the need for security measures that cover all interaction modes with LLMs, including tool use, to prevent emergent jailbreak vectors in real-world deployments.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities, but their power comes with significant security considerations. While extensive research has been conducted on the safety of LLMs in chat mode, the security implications of their function calling feature have been largely overlooked. This paper uncovers a critical vulnerability in the function calling process of LLMs, introducing a novel "jailbreak function" attack method that exploits alignment discrepancies, user coercion, and the absence of rigorous safety filters. Our empirical study, conducted on six state-of-the-art LLMs including GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-pro, reveals an alarming average success rate of over 90\% for this attack. We provide a comprehensive analysis of why function calls are susceptible to such attacks and propose defensive strategies, including the use of defensive prompts. Our findings highlight the urgent need for enhanced security measures in the function calling capabilities of LLMs, contributing to the field of AI safety by identifying a previously unexplored risk, designing an effective attack method, and suggesting practical defensive measures. Our code is available at https://github.com/wooozihui/jailbreakfunction.
Paper Structure (8 sections, 3 figures, 3 tables)

This paper contains 8 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Jailbreak Function
  4. Empirical Studies
  5. Assessing the Effectiveness of Jailbreak Functions (RQ 1)
  6. Analyzing Why Function Calls Cause Jailbreaks (RQ 2)
  7. Discussion of the Defense Strategy (RQ 3)
  8. Conclusion

Figures (3)

  • Figure 1: Overview of the function calling process in LLMs and the potential for jailbreak attacks.
  • Figure 2: Components of the jailbreak function calling.
  • Figure 3: Screenshot showcasing generation of harmful content: GPT-4o (Left), Claude-3.5-Sonnet (Center), Gemini-1.5-Pro (Right).