Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity

TL;DR

The paper introduces ETAPP, a benchmark for evaluating personalized tool invocation in LLMs from the perspectives of personalization and proactivity. It builds a sandboxed environment with 33 APIs across 9 categories and constructs a dataset of 800 testing cases derived from 16 user profiles and 9 days of interaction history, complemented by manually annotated key points for evaluation. A key-point-based LLM evaluation framework is proposed to improve reliability over direct judgments. Through extensive experiments, the authors analyze tool-invoking methods, preference settings, and fine-tuning, finding that enhanced reasoning before tool usage improves personalization and proactivity, while fine-tuning yields in-domain gains with limited out-of-domain benefits. The work provides a publicly available dataset and code, advancing research on personalized tool-augmented LLM agents.

Abstract

Personalized tool utilization is essential for aligning large language models (LLMs) with user preference in interaction scenarios with various tools. However, most of the current benchmarks primarily focus on either personalization of text generation or direct tool-utilizing, without considering both. In this work, we introduce a novel benchmark ETAPP for evaluating personalized tool invocation, establishing a sandbox environment, and a comprehensive dataset of 800 testing cases covering diverse user profiles. To improve the accuracy of our evaluation, we propose a key-point-based LLM evaluation method, mitigating biases in the LLM-as-a-judge system by manually annotating key points for each test case and providing them to LLM as the reference. Additionally, we evaluate the excellent LLMs and provide an in-depth analysis. Furthermore, we investigate the impact of different tool-invoking strategies on LLMs' personalization performance and the effects of fine-tuning in our task. The effectiveness of our preference-setting and key-point-based evaluation method is also validated. Our findings offer insights into improving personalized LLM agents. Our Code is available at https://github.com/hypasd-art/ETAPP.

Figures (10)

• Figure 1: The difference between a traditional tool-augmented agent and a personal tool-augmented agent. Red font represents output reflecting personalization, while yellow background font represents output reflecting proactivity.
• Figure 2: The process of dataset construction.
• Figure 3: The process of Inference and Evaluation of our benchmark.
• Figure 4: The performance of different tool-invoking methods.
• Figure 5: The Bland-Altman analysis result of Proactivity with given key points.