Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answering

Nil-Jana Akpinar, Chia-Jung Lee, Vanessa Murdock, Pietro Perona

TL;DR

This work presents the first systematic evaluation of LLM robustness to inquiry personas in factual QA, showing that first-person user attributes can significantly alter answers, trigger refusals, or induce role confusion. By constructing a rigorous framework with objective QA datasets, diverse persona types, and multi-model evaluation, the study reveals both robustness gaps and scenarios where persona alignment can improve accuracy. It introduces a push-button robustness testing approach using simple system prompts to mitigate persona-induced variability and provides deep thematic analysis of failure modes to inform mitigation. The findings highlight the need for careful handling of user context in real-world QA systems and offer strategies for improving reliability, such as objective prompts, persona cue filtering, and targeted adversarial training.

Abstract

Large Language Models (LLMs) should answer factual questions truthfully, grounded in objective knowledge, regardless of user context such as self-disclosed personal information, or system personalization. In this paper, we present the first systematic evaluation of LLM robustness to inquiry personas, i.e. user profiles that convey attributes like identity, expertise, or belief. While prior work has primarily focused on adversarial inputs or distractors for robustness testing, we evaluate plausible, human-centered inquiry persona cues that users disclose in real-world interactions. We find that such cues can meaningfully alter QA accuracy and trigger failure modes such as refusals, hallucinated limitations, and role confusion. These effects highlight how model sensitivity to user framing can compromise factual reliability, and position inquiry persona testing as an effective tool for robustness evaluation.

Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answering

TL;DR

This work presents the first systematic evaluation of LLM robustness to inquiry personas in factual QA, showing that first-person user attributes can significantly alter answers, trigger refusals, or induce role confusion. By constructing a rigorous framework with objective QA datasets, diverse persona types, and multi-model evaluation, the study reveals both robustness gaps and scenarios where persona alignment can improve accuracy. It introduces a push-button robustness testing approach using simple system prompts to mitigate persona-induced variability and provides deep thematic analysis of failure modes to inform mitigation. The findings highlight the need for careful handling of user context in real-world QA systems and offer strategies for improving reliability, such as objective prompts, persona cue filtering, and targeted adversarial training.

Abstract

Large Language Models (LLMs) should answer factual questions truthfully, grounded in objective knowledge, regardless of user context such as self-disclosed personal information, or system personalization. In this paper, we present the first systematic evaluation of LLM robustness to inquiry personas, i.e. user profiles that convey attributes like identity, expertise, or belief. While prior work has primarily focused on adversarial inputs or distractors for robustness testing, we evaluate plausible, human-centered inquiry persona cues that users disclose in real-world interactions. We find that such cues can meaningfully alter QA accuracy and trigger failure modes such as refusals, hallucinated limitations, and role confusion. These effects highlight how model sensitivity to user framing can compromise factual reliability, and position inquiry persona testing as an effective tool for robustness evaluation.
Paper Structure (19 sections, 5 figures, 5 tables)

This paper contains 19 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Question Answering and Objectivity
  4. Prompt Sensitivity And Robustness
  5. Personas And Stereotyping
  6. Methodology
  7. Inquiry Personas
  8. Objective Question-Answering Datasets
  9. Language Models
  10. Metrics
  11. Thematic Analysis
  12. Results
  13. Preliminary Robustness Interventions
  14. Accuracy Robustness to Main Personas
  15. Thematic Analysis of Failure Modes for Unaligned Personas
  16. ...and 4 more sections

Figures (5)

  • Figure 1: A simple example of how an inquiry persona affects the correctness of model responses in factual question answering.
  • Figure 2: Overview of LLM robustness testing framework by inquiry personas. Factual question-answering datasets and diverse personas are prepared (left), robustness testing is conducted by plugging in desired candidate LLMs under the pre-set system prompt and user prompt template (center), and then multi-faceted evaluation and analysis are performed (right).
  • Figure 3: The effect of including or excluding the system prompt (focusing on factual question-answering) for robustness testing. Each point reflects the accuracy percentage change compared to no persona in no system prompt and with system prompt settings respectively.
  • Figure 4: Absolute accuracy (upper) and accuracy change rate in percentage (lower) across persona types, models and datasets. $\blacktriangle$ and $\blacktriangledown$ represent significant differences compared to the no persona with $p<0.05$ using McNemar's test. Bold text represents the highest accuracy row-wise. The presence of inquiry personas can impact the robustness of LLMs with varying degrees depending on the models, persona types, and datasets.
  • Figure 5: LLM-based metrics for different types of fine-grained personas in TriviaQA using Sonnet 3 (first row), Haiku 3.5 (second row), and Sonnet 3.5 (third row) models. Eight metrics are adopted, including Language Complexity (LC), Structured Format (SF), Illustrative Examples (IE), References (RF), Limitation Acknowledgment (LA), Expertise Match (EM), Context Relevance (CR), and Stereotype Risk (SR). Results show variations across persona types and models.