Phare: A Safety Probe for Large Language Models

TL;DR

Phare introduces a multilingual diagnostic framework to systematically probe LLM safety along three axes: hallucination, biases, and harmful content. Unlike ranking benchmarks, Phare focuses on exposing specific failure modes through a multilingual dataset, three dedicated modules, and a robust human-plus-LLM evaluation protocol, with public data and code for reproducibility. Evaluating 17 leading LLMs reveals pervasive vulnerabilities such as sycophancy, prompt sensitivity, and stereotype propagation, while also demonstrating strong harm-prevention performance across providers, particularly for explicit harmful requests. The work highlights practical implications for deploying safer, better-aligned LLMs and outlines clear directions for expanding language coverage, adding modules, and exploring reasoning-enabled models in future research.

Abstract

Ensuring the safety of large language models (LLMs) is critical for responsible deployment, yet existing evaluations often prioritize performance over identifying failure modes. We introduce Phare, a multilingual diagnostic framework to probe and evaluate LLM behavior across three critical dimensions: hallucination and reliability, social biases, and harmful content generation. Our evaluation of 17 state-of-the-art LLMs reveals patterns of systematic vulnerabilities across all safety dimensions, including sycophancy, prompt sensitivity, and stereotype reproduction. By highlighting these specific failure modes rather than simply ranking models, Phare provides researchers and practitioners with actionable insights to build more robust, aligned, and trustworthy language systems.

Figures (24)

• Figure 1: Phare dataset generation and LLMs evaluation methodology.
• Figure 2: Impact of prompt and input perturbations on hallucination-related tasks. A. Effect of user message confidence tone on model ability to debunk controversial claims. Each cell shows the average debunking accuracy score, and the models are sorted by increasing p-value for the $\chi^2$ test. Details about the statistics are reported in Appendix \ref{['app:chi2_hallucination']}. B. Impact of system prompt instructions (neutral vs. concise formulation) on model resistance to misinformation (see Appendix \ref{['app:chi2_hallucination']} for statistical details). C. Tool call accuracy under different input perturbations. Bars represent mean accuracy across all evaluated models, with error bars indicating 95% confidence intervals.
• Figure 3: A. Generation pipeline for measuring attribute associations in open-ended generation tasks. B. Cramér's V association measure between base and extracted attributes, across stories generated by all models. C. Proportion of models achieving good self-coherency score (> 0.7) by base attribute. D. Examples of debatable associations and real-world patterns.
• Figure 4: Resistance to harmful misguidance across all tested models.
• Figure 5: Scores per category, task and language for hallucinations aggregated over models. Overall there's a performance variability on language but not necessarily consistent over the submodules, except for English which is most of the time better handled by LLMs.