A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models

TL;DR

The paper addresses the problem that existing LLM benchmarks emphasize model abilities rather than how well models satisfy real user intents across diverse cultures. It introduces URS, a dataset of 1,846 authentic user interactions (1,014 English and 832 Chinese) across six intents collected from 712 participants in 23 countries, and evaluates 10 LLM services using GPT-4 as an intent-aware judge. Findings show strong alignment between benchmark scores and human preferences (Pearson r = 0.95 with user satisfaction and r = 0.94 with pairwise annotations); GPT-4 generally yields the highest performance, with exceptions in the Text Assistant domain. The work provides a scalable, user-centric benchmark and makes the URS dataset and process publicly available to guide LLM development and service selection.

Abstract

Large language models (LLMs) are essential tools that users employ across various scenarios, so evaluating their performance and guiding users in selecting the suitable service is important. Although many benchmarks exist, they mainly focus on specific predefined model abilities, such as world knowledge, reasoning, etc. Based on these ability scores, it is hard for users to determine which LLM best suits their particular needs. To address these issues, we propose to evaluate LLMs from a user-centric perspective and design this benchmark to measure their efficacy in satisfying user needs under distinct intents. Firstly, we collect 1,846 real-world use cases from a user study with 712 participants from 23 countries. This first-hand data helps us understand actual user intents and needs in LLM interactions, forming the User Reported Scenarios (URS) dataset, which is categorized with six types of user intents. Secondly, based on this authentic dataset, we benchmark 10 LLM services with GPT-4-as-Judge. Thirdly, we show that benchmark scores align well with human preference in both real-world experience and pair-wise annotations, achieving Pearson correlations of 0.95 and 0.94, respectively. This alignment confirms that the URS dataset and our evaluation method establish an effective user-centric benchmark. The dataset, code, and process data are available at https://github.com/Alice1998/URS.

Figures (10)

• Figure 1: Existing benchmarks are mainly model ability-focused and categorized by tasks chang2023survey. We benchmark LLMs on User Reported Scenarios (URS), which are user-centric, intent-driven, multi-cultural, and multi-LLM usage cases involved.
• Figure 2: IP Distribution of the 712 participants.
• Figure 3: Evaluation Procedure. For each evaluation instance, the evaluator is provided with the user intent, five intent-aware criteria, chain-of-though reasoning steps, scoring standards for each two-point segment, addition with the question, an 8-score reference answer for this question, and the test LLM output for evaluation. Then, a parser will extract the final score from the evaluator's detailed rating content to form the benchmark.
• Figure 4: Comparison between GPT-4 and Claude-3. "A Eva B Ans" indicates the evaluation setting, where 'A' denotes the evaluator LLM and 'B' represents the source LLM used to generate 8-point reference answers. These results show that GPT-4 slightly outperforms Claude-3.
• Figure 5: Benchmark Score and User Reported Satisfaction Correlate Well across Intents. "Benchmark Score" is averaged under different intents. "User Reported Satisfaction" is the average satisfaction level reported in the user study. Intents are ranked by user satisfaction.