Is Passive Expertise-Based Personalization Enough? A Case Study in AI-Assisted Test-Taking
Li Siyan, Jason Zhang, Akash Maharaj, Yuanming Shi, Yunyao Li
TL;DR
The paper investigates passive expertise-based personalization in an enterprise AI assistant, implementing a framework that estimates user expertise, classifies query domain, and adapts responses via prompt-based content and stylistic controls. A within-subject user study with timed exam questions compares a baseline assistant to a passively personalized one, revealing reduced task load and improved perceived assistance for some users, but increased mental load under time pressure for non-experts. Key contributions include an extensible passive personalization framework, empirical evidence on novice-expert differences, and insights into the need for user agency to complement passive personalization. The findings highlight that combining active and passive personalization can better optimize user experience and task effectiveness in knowledge-intensive, enterprise settings.
Abstract
Novice and expert users have different systematic preferences in task-oriented dialogues. However, whether catering to these preferences actually improves user experience and task performance remains understudied. To investigate the effects of expertise-based personalization, we first built a version of an enterprise AI assistant with passive personalization. We then conducted a user study where participants completed timed exams, aided by the two versions of the AI assistant. Preliminary results indicate that passive personalization helps reduce task load and improve assistant perception, but reveal task-specific limitations that can be addressed through providing more user agency. These findings underscore the importance of combining active and passive personalization to optimize user experience and effectiveness in enterprise task-oriented environments.