ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models
Zihao Cheng, Hongru Wang, Zeming Liu, Yuhang Guo, Yuanfang Guo, Yunhong Wang, Haifeng Wang
TL;DR
ToolSpectrum tackles the challenge of context-aware personalization in tool-augmented LLMs by formalizing personalized tool utilization around user profiles and environmental factors, and by constructing the first benchmark that jointly considers overlapping toolsets and domain policies. The authors design a comprehensive data collection pipeline across nine domains, building toolsets, profiles, and environments, and generate tool-call results with GPT-4o to evaluate how LLMs reason about personalized tool selection. Experimental results show that personalization significantly improves tool utilization, but current models struggle to jointly reason about multiple personalization dimensions, with performance dropping as more factors are included. The work highlights the need for context-aware personalization in tool-enabled LLMs and provides a foundation for future research and benchmark-driven progress in personalized tool usage.
Abstract
While integrating external tools into large language models (LLMs) enhances their ability to access real-time information and domain-specific services, existing approaches focus narrowly on functional tool selection following user instructions, overlooking the context-aware personalization in tool selection. This oversight leads to suboptimal user satisfaction and inefficient tool utilization, particularly when overlapping toolsets require nuanced selection based on contextual factors. To bridge this gap, we introduce ToolSpectrum, a benchmark designed to evaluate LLMs' capabilities in personalized tool utilization. Specifically, we formalize two key dimensions of personalization, user profile and environmental factors, and analyze their individual and synergistic impacts on tool utilization. Through extensive experiments on ToolSpectrum, we demonstrate that personalized tool utilization significantly improves user experience across diverse scenarios. However, even state-of-the-art LLMs exhibit the limited ability to reason jointly about user profiles and environmental factors, often prioritizing one dimension at the expense of the other. Our findings underscore the necessity of context-aware personalization in tool-augmented LLMs and reveal critical limitations for current models. Our data and code are available at https://github.com/Chengziha0/ToolSpectrum.