ConsistencyAI: A Benchmark to Assess LLMs' Factual Consistency When Responding to Different Demographic Groups

TL;DR

ConsistencyAI introduces an independent benchmark to quantify how LLMs vary their factual outputs across demographic personas. By prompting 19 models over 15 topics and across 100 randomized personas, the study uses sentence-embedding representations and cross-persona cosine similarity to derive a factual consistency score, with a mean benchmark of $0.8656$. The results reveal strong overall consistency but meaningful variation across providers and topics, showing that newer or more capable models do not necessarily improve consistency, and that controversial or dynamic domains are more challenging. The paper also provides an interactive webapp and open-source pipeline to enable reproducible, cross-model analysis and promotes persona-invariant prompting strategies to mitigate demographic bias. These findings have policy relevance for ideological neutrality, support for independent benchmarking, and practical implications for developers aiming to deliver reliable, audience-agnostic factual information from LLMs.

Abstract

Is an LLM telling you different facts than it's telling me? This paper introduces ConsistencyAI, an independent benchmark for measuring the factual consistency of large language models (LLMs) for different personas. ConsistencyAI tests whether, when users of different demographics ask identical questions, the model responds with factually inconsistent answers. Designed without involvement from LLM providers, this benchmark offers impartial evaluation and accountability. In our experiment, we queried 19 LLMs with prompts that requested 5 facts for each of 15 topics. We repeated this query 100 times for each LLM, each time adding prompt context from a different persona selected from a subset of personas modeling the general population. We processed the responses into sentence embeddings, computed cross-persona cosine similarity, and computed the weighted average of cross-persona cosine similarity to calculate factual consistency scores. In 100-persona experiments, scores ranged from 0.9065 to 0.7896, and the mean was 0.8656, which we adopt as a benchmark threshold. xAI's Grok-3 is most consistent, while several lightweight models rank lowest. Consistency varies by topic: the job market is least consistent, G7 world leaders most consistent, and issues like vaccines or the Israeli-Palestinian conflict diverge by provider. These results show that both the provider and the topic shape the factual consistency. We release our code and interactive demo to support reproducible evaluation and encourage persona-invariant prompting strategies.

Figures (6)

• Figure 1: The basic pipeline of our benchmark.
• Figure 2: Models disproportionately failed to respond to queries about the Israeli-Palestinian Conflict (first-pass).
• Figure 3: Deepseek-Chat-v3 was approximately twice as nonresponsive as the average model when asked about the Israeli-Palestinian Conflict (first-pass).
• Figure 4: Age distribution of the personas used in this experiment
• Figure 5: Personality characteristics represented in experimental personas.