LongFuncEval: Measuring the effectiveness of long context models for function calling

TL;DR

Long-context tool calling presents practical challenges for enterprise AI assistants. The paper introduces three challenges—tool catalog scale, long tool responses, and long multi-turn conversations—and constructs datasets and evaluation setups to study them. It benchmarks several 128K-context LLMs and shows substantial degradation as context grows across catalog size, response length, and the number of turns. The work highlights the need for model and dataset innovations to improve reliability of function calling in long contexts.

Abstract

Multiple recent studies have documented large language models' (LLMs) performance on calling external tools/functions. Others focused on LLMs' abilities to handle longer context lengths. At the intersection of these areas lies another interesting problem: LLMs' abilities to accurately perform function calls in long context settings. Particularly, when calling tools, LLMs are encumbered by three predominant challenges: (1) a large catalog of tools, (2) long responses from the tool APIs, and (3) long multi-turn conversations. These challenges are particularly relevant to enterprise applications of LLMs which engage in multi-turn conversations with users to complete complex tasks that require a large catalog of complex tools. The literature contains multiple investigations of long context challenges such as lost in the middle or needle in the haystack for natural language tasks. In this paper, we make the first attempt to comprehensively study the long context understanding capabilities of these models in the tool calling setup. We modify existing benchmarks for challenge 1 and 3, and create a new evaluation set for challenge 2 to enable this analysis. We gradually increase the input context length and also vary the position of the answer in the input. When evaluated with several long context models, we observe a performance drop of 7% to 85% as the number of tools increases, a 7% to 91% degradation in answer retrieval as the tool responses length increases, and 13% and 40% degradation for as multi-turn conversations get longer. Our study shows that LLMs still struggle with long context in tool calling settings, motivating future research to drive further LLM improvements.

Figures (9)

• Figure 1: Overview of the three challenges at the intersection of long context and tool calling
• Figure 2: Examples of tool response question categories for challenge 2
• Figure 3: Challenge 1: AST accuracy for live_simple. The performance degradation for larger contexts can be clearly seen as we move from left to right in each plot. For larger contexts, we can also notice a recency bias with the higher values of positions performing better.
• Figure 4: Challenge 2: Performance comparison (a) when tool response token limit is increased from 10K to 80K for position 1, (b) when position is varied from 1 to 8 for token limit of 80K. The performance degradation due to large responses is clearly seen in (a), whereas (b) shows recency bias.
• Figure 5: Challenge 1: average AST accuracy for all datasets