Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluation

TL;DR

CoRUS introduces a framework to simulate role-based user questions for evaluating conversational AI in stigmatized domains, focusing on opioid use disorder recovery. It builds a taxonomy of asker roles (Patient, Caregiver, Practitioner) from r/OpiatesRecovery, curates seed questions, and generates 15,321 role-aware queries to probe LLM behavior. Across five LLMs, role context systematically shifts responses: patients and caregivers elicit more supportive and readable outputs with less knowledge content, while practitioner framing maintains knowledge but reduces support and readability; abstentions rise for vulnerable roles. The work demonstrates the necessity of role-aware evaluation for aligning AI behavior with diverse user needs and provides a scalable, privacy-conscious methodology extendable to other domains, while acknowledging limitations in data source scope and reliance on automatic evaluators.

Abstract

Language model users often embed personal and social context in their questions. The asker's role -- implicit in how the question is framed -- creates specific needs for an appropriate response. However, most evaluations, while capturing the model's capability to respond, often ignore who is asking. This gap is especially critical in stigmatized domains such as opioid use disorder (OUD), where accounting for users' contexts is essential to provide accessible, stigma-free responses. We propose CoRUS (COmmunity-driven Roles for User-centric Question Simulation), a framework for simulating role-based questions. Drawing on role theory and posts from an online OUD recovery community (r/OpiatesRecovery), we first build a taxonomy of asker roles -- patients, caregivers, practitioners. Next, we use it to simulate 15,321 questions that embed each role's goals, behaviors, and experiences. Our evaluations show that these questions are both highly believable and comparable to real-world data. When used to evaluate five LLMs, for the same question but differing roles, we find systematic differences: vulnerable roles, such as patients and caregivers, elicit more supportive responses (+17%) and reduced knowledge content (-19%) in comparison to practitioners. Our work demonstrates how implicitly signaling a user's role shapes model responses, and provides a methodology for role-informed evaluation of conversational AI.

Figures (11)

• Figure 1: CoRUS framework. Users' framing of a question often reflects the social context in which it is asked, yet most evaluations ignore this. We propose CoRUS to (1) derive a taxonomy of askers' roles, grounded in role theory and posts from an online recovery community (§\ref{['sec:taxonomy']}), (2) simulate role-based questions embedded with social context (§\ref{['sec:simulation']}), and (3) evaluate models to assess how role-based context influences response (§\ref{['sec:method-audit']}).
• Figure 2: Evaluation of Simulated Role-based Questions. Top: CoRUS produces questions with believability comparable to real-world ones. Bottom: CoRUS produces significantly more human-like and contextually plausible queries than other prompting methods, while maintaining high interaction plausibility and other properties (Appendix \ref{['sec:appendix-sim-eval']}, Figure \ref{['fig:appendix-corus-eval']}). * denotes significant difference from CoRUS (Chi-square test, $p<0.05$); error bars show $95\%$ CI.
• Figure 3: Percentage change in Knowledge, Support and Readability scores of model responses when role-based context is added to the queries. Across all models, patient and caregiver context reduces knowledge, but increases support and readability. Practitioner context shows the inverse trend, with lower support, higher readability, and no significant change in knowledge. More details in Figures \ref{['fig:llm-eval-context']} and \ref{['fig:box-plot-llm-eval']}. Error bars represent 95% CI.
• Figure 4: Abstention. Proportion of role-agnostic and role-based queries that models refuse to answer.
• Figure 5: Evaluation of clusters using Silhouette Analysis.