RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models
Tianhao Shen, Sun Li, Quan Tu, Deyi Xiong
TL;DR
RoleEval addresses the need for systematic evaluation of LLMs' role knowledge across languages. It introduces a bilingual benchmark with 6,000 parallel MCQs over 300 characters (RoleEval-Global and RoleEval-Chinese), emphasizing memorization, utilization, and multi-hop reasoning about personal attributes, relationships, and experiences. The authors implement a hybrid quality-check process and evaluate a wide range of open-source and proprietary LLMs under zero-/few-shot settings, revealing that GPT-4 excels on RoleEval-Global while Chinese LLMs lead on RoleEval-Chinese, highlighting language-specific knowledge distributions. The work underscores the importance of cross-lingual and culture-aware evaluation and discusses scaling laws and future directions to improve role knowledge in multilingual LLMs.
Abstract
The rapid evolution of large language models necessitates effective benchmarks for evaluating their role knowledge, which is essential for establishing connections with the real world and providing more immersive interactions. This paper introduces RoleEval, a bilingual benchmark designed to assess the memorization, utilization, and reasoning capabilities of role knowledge. RoleEval comprises RoleEval-Global (including internationally recognized characters) and RoleEval-Chinese (including characters popular in China), with 6,000 Chinese-English parallel multiple-choice questions focusing on 300 influential people and fictional characters drawn from a variety of domains including celebrities, anime, comics, movies, TV series, games, and fictions. These questions cover basic knowledge and multi-hop reasoning abilities, aiming to systematically probe various aspects such as personal information, relationships, abilities, and experiences of the characters. To maintain high standards, we perform a hybrid quality check process combining both automatic and human verification, ensuring that the questions are diverse, challenging, and discriminative. Our extensive evaluations with RoleEval across various open-source and proprietary large language models, under both the zero- and few-shot settings, reveal insightful findings. Notably, while GPT-4 outperforms other models on RoleEval-Global, Chinese large language models excel on RoleEval-Chinese, highlighting significant knowledge distribution differences. We expect that RoleEval would highlight the significance of assessing role knowledge for large language models across various languages and cultural settings.