How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO

TL;DR

This work introduces ECHO, a Turing-test-inspired framework to evaluate LLMs' ability to role-play ordinary individuals by leveraging acquaintances of real people as judges. It systematically constructs role-playing LLMs from real-person profiles, collects paired human and machine responses, and assesses deception performance across multiple baselines, finding that GPT-4-Turbo and GPTs deliver strongest impersonation (GPTs reaching $48.3\%$ overall). The study also probes LLMs as evaluators, showing GPT-4 family models can distinguish human- vs machine-generated text with high accuracy but display instruction biases, while Gemini approaches near random guessing and length bias is minimal. Together, results demonstrate both the potential of LLM-based digital clones for NPC-like applications and the need to account for evaluator biases in AI testing, with important ethical and data-protection considerations. ECHO provides a scalable, open framework for comparing role-playing methods and advancing understanding of LLM mimicry in real-world, non-celebrity contexts.

Abstract

The role-play ability of Large Language Models (LLMs) has emerged as a popular research direction. However, existing studies focus on imitating well-known public figures or fictional characters, overlooking the potential for simulating ordinary individuals. Such an oversight limits the potential for advancements in digital human clones and non-player characters in video games. To bridge this gap, we introduce ECHO, an evaluative framework inspired by the Turing test. This framework engages the acquaintances of the target individuals to distinguish between human and machine-generated responses. Notably, our framework focuses on emulating average individuals rather than historical or fictional figures, presenting a unique advantage to apply the Turing Test. We evaluated three role-playing LLMs using ECHO, with GPT-3.5 and GPT-4 serving as foundational models, alongside the online application GPTs from OpenAI. Our results demonstrate that GPT-4 more effectively deceives human evaluators, and GPTs achieves a leading success rate of 48.3%. Furthermore, we investigated whether LLMs could discern between human-generated and machine-generated texts. While GPT-4 can identify differences, it could not determine which texts were human-produced. Our code and results of reproducing the role-playing LLMs are made publicly available via https://github.com/CUHK-ARISE/ECHO.

Figures (2)

• Figure 1: Success rates of role-playing LLMs in deceiving human evaluators. The human evaluators are instructed to identify human-generated responses.
• Figure 2: An illustration of the design of ECHO.