SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinations

TL;DR

SECA addresses the vulnerability of LLMs to hallucinations triggered by semantically equivalent and coherent prompt variations. It models attack generation as a constrained optimization, leveraging a semantic-equivalence proposer and a feasibility checker to produce realistic prompts that elicit targeted hallucinations. Experiments on MMLU with multiple LLMs show SECA achieves higher attack success while maintaining semantic equivalence and coherence; analysis reveals verbose, lexically diverse prompts are more effective. The work provides a framework for realistic robustness evaluation and releases code for replication.

Abstract

Large Language Models (LLMs) are increasingly deployed in high-risk domains. However, state-of-the-art LLMs often produce hallucinations, raising serious concerns about their reliability. Prior work has explored adversarial attacks for hallucination elicitation in LLMs, but it often produces unrealistic prompts, either by inserting gibberish tokens or by altering the original meaning. As a result, these approaches offer limited insight into how hallucinations may occur in practice. While adversarial attacks in computer vision often involve realistic modifications to input images, the problem of finding realistic adversarial prompts for eliciting LLM hallucinations has remained largely underexplored. To address this gap, we propose Semantically Equivalent and Coherent Attacks (SECA) to elicit hallucinations via realistic modifications to the prompt that preserve its meaning while maintaining semantic coherence. Our contributions are threefold: (i) we formulate finding realistic attacks for hallucination elicitation as a constrained optimization problem over the input prompt space under semantic equivalence and coherence constraints; (ii) we introduce a constraint-preserving zeroth-order method to effectively search for adversarial yet feasible prompts; and (iii) we demonstrate through experiments on open-ended multiple-choice question answering tasks that SECA achieves higher attack success rates while incurring almost no semantic equivalence or semantic coherence errors compared to existing methods. SECA highlights the sensitivity of both open-source and commercial gradient-inaccessible LLMs to realistic and plausible prompt variations. Code is available at https://github.com/Buyun-Liang/SECA.

Figures (10)

• Figure 1: Our SECA finds semantically equivalent and coherent attacks to elicit LLM hallucinations. See Appendix \ref{['app:ex_attack_response']} for a detailed example.
• Figure 2: Starting from the current prompt $\bm{x}_k$, SECA generates the next prompt $\bm{x}_{k+1}$ while enforcing semantic equivalence and coherence constraints.
• Figure 3: (Left) The values of ASR@30 of Raw (dotted lines) and SECA (solid lines). (Middle) Average difference between the objective values $\Delta \log P(y^*|\bm{x})$ of Raw and SECA. (Right) Average SCE and SEE of SECA. GCG semantic errors are shown for reference, sourced from \ref{['tab:summary_avg_results']}. Evaluations are on a filtered MMLU subset across 16 subjects (see \ref{['sec:setup']}). Please see Appendix §\ref{['app:raw_data']} for the data used in the plots.
• Figure 4: Analysis of SECA: (a) objective progression over iterations; (b) hallucination type breakdown; and (c) lexical diversity and verbosity of SECA prompts. The shaded area in (a) and the error bar in (c) represent the standard deviation calculated over 10,000 bootstrap samples.
• Figure 5: Illustration of a factuality hallucination induced by a SECA adversarial prompt. The top two green boxes show the full attack prompt based on the original MMLU question in elementary mathematics, followed by the faithful and factual response from the target LLM. The bottom two blue boxes present a SECA-generated adversarial variant of the original prompt, with edits highlighted in red, and the corresponding target LLM explanation, which includes red-highlighted hallucinated content. In this example, the model selects the incorrect choice ('B') and generates a hallucinated explanation, showcasing a factuality hallucination.