Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Weaknesses in Function Call Models via Reinforcement Learning: An Adversarial Data Augmentation Approach

Weiran Guo, Bing Bo, Shaoxiang Wu, Jingsheng Yang

TL;DR

The paper tackles robustness gaps in function calling for LLMs by introducing a reinforcement-learning–driven adversarial data augmentation framework that rewrites seed data to reveal FC weaknesses. It models a zero-sum interaction between a query model $\pi_Q$ and an FC model $\pi_F$, using a two-stage reward design and curriculum-guided iterative training to generate targeted, diverse adversarial prompts. Key contributions include a two-stage filtering mechanism, an adversarial reward, embedding-distance diversity regularization, and an iterative alternating training scheme that progressively increases task difficulty. Empirical results on BFCL across multiple base models show improved FC accuracy and robustness, especially for smaller models, indicating the method's practicality for real-world tool-use scenarios and its potential applicability beyond function calling.

Abstract

Function call capabilities have become crucial for Large Language Models (LLMs), enabling them to interact more effectively with external tools and APIs. Existing methods for improving the function call capabilities of LLMs rely on data obtained either through manual annotation or automated generation by models, and use this data to finetune the LLMs. However, these methods often lack targeted design and are constrained by fixed patterns and data distributions, which limits their effectiveness in enhancing the generalization and robustness of function call LLMs. To address this limitation, we propose a novel adversarial data augmentation method that employs reinforcement learning to systematically identify and target the weaknesses of function call LLMs. Our training framework introduces a query model trained with reinforcement learning (RL) to generate adversarial queries that are specifically designed to challenge function call (FC) models. This approach adopts a zero sum game formulation, where the query model and the FC model engage in iterative alternating training. Overall, our method advances the development of more robust FC models and provides a systematic way to identify and correct weaknesses in the ability of LLMs to interact with external tools.

Exploring Weaknesses in Function Call Models via Reinforcement Learning: An Adversarial Data Augmentation Approach

TL;DR

The paper tackles robustness gaps in function calling for LLMs by introducing a reinforcement-learning–driven adversarial data augmentation framework that rewrites seed data to reveal FC weaknesses. It models a zero-sum interaction between a query model and an FC model , using a two-stage reward design and curriculum-guided iterative training to generate targeted, diverse adversarial prompts. Key contributions include a two-stage filtering mechanism, an adversarial reward, embedding-distance diversity regularization, and an iterative alternating training scheme that progressively increases task difficulty. Empirical results on BFCL across multiple base models show improved FC accuracy and robustness, especially for smaller models, indicating the method's practicality for real-world tool-use scenarios and its potential applicability beyond function calling.

Abstract

Function call capabilities have become crucial for Large Language Models (LLMs), enabling them to interact more effectively with external tools and APIs. Existing methods for improving the function call capabilities of LLMs rely on data obtained either through manual annotation or automated generation by models, and use this data to finetune the LLMs. However, these methods often lack targeted design and are constrained by fixed patterns and data distributions, which limits their effectiveness in enhancing the generalization and robustness of function call LLMs. To address this limitation, we propose a novel adversarial data augmentation method that employs reinforcement learning to systematically identify and target the weaknesses of function call LLMs. Our training framework introduces a query model trained with reinforcement learning (RL) to generate adversarial queries that are specifically designed to challenge function call (FC) models. This approach adopts a zero sum game formulation, where the query model and the FC model engage in iterative alternating training. Overall, our method advances the development of more robust FC models and provides a systematic way to identify and correct weaknesses in the ability of LLMs to interact with external tools.
Paper Structure (21 sections, 7 equations, 2 figures, 2 tables, 1 algorithm)

This paper contains 21 sections, 7 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Function Calling in LLMs
  4. Two-Player RL-based Training in LLMs
  5. Priliminaries
  6. Problem Setting
  7. LLM Post-Training
  8. Zero-Sum Game
  9. Bad Case Exploring: Query Model
  10. Data Preparation
  11. Reward Design
  12. Two-stage Filtering Mechanism
  13. Adversarial Reward
  14. Diversity: Embedding Loss
  15. Early Stop
  16. ...and 6 more sections

Figures (2)

  • Figure 1: The illustration of a function calling process.
  • Figure 2: The complete process of one iteration of training.