StealthRank: LLM Ranking Manipulation via Stealthy Prompt Optimization

TL;DR

StealthRank investigates stealthy adversarial manipulations of LLM-based ranking by injecting optimized prompts into item descriptions. It introduces an energy-based SRP objective, optimized with Langevin dynamics in logit space to balance ranking gains, linguistic fluency, and avoidance of obvious promotional cues. Across four instruction-tuned LLM rerankers and two datasets, SRP achieves stronger target promotion while maintaining natural language and minimal detectable signals, with ablations and human studies corroborating its effectiveness. The work highlights security vulnerabilities in LLM-driven retrieval systems and motivates defenses to strengthen robustness in practical product search and document retrieval pipelines.

Abstract

The integration of large language models (LLMs) into information retrieval systems introduces new attack surfaces, particularly for adversarial ranking manipulations. We present $\textbf{StealthRank}$, a novel adversarial attack method that manipulates LLM-driven ranking systems while maintaining textual fluency and stealth. Unlike existing methods that often introduce detectable anomalies, StealthRank employs an energy-based optimization framework combined with Langevin dynamics to generate StealthRank Prompts (SRPs)-adversarial text sequences embedded within item or document descriptions that subtly yet effectively influence LLM ranking mechanisms. We evaluate StealthRank across multiple LLMs, demonstrating its ability to covertly boost the ranking of target items while avoiding explicit manipulation traces. Our results show that StealthRank consistently outperforms state-of-the-art adversarial ranking baselines in both effectiveness and stealth, highlighting critical vulnerabilities in LLM-driven ranking systems. Our code is publicly available at $\href{https://github.com/Tangyiming205069/controllable-seo}{here}$.

Figures (4)

• Figure 1: Overview of the LLM-based ranking attack pipeline. Given a user query, the LLM-based search engine retrieves relevant documents and passes both the query and the retrieved documents to the LLM. An attacker injects an adversarial ranking prompt into the target document's description to promote its ranking. The manipulated list is then processed by the LLM, resulting in a response where the target document is ranked higher while the ranking prompt remains stealthy.
• Figure 1: Performance comparison among our method (SRP), TAP, and STS across four models on three evaluation metrics.
• Figure 2: Overview of StealthRank. (1) An initial prompt $\tilde{\mathbf{y}}^{(0)}$ is formed by merging the target product description with a guiding sentence (§\ref{['sec:srp']}). (2) The product list and user query are passed to the LLM-based system. (3) Langevin dynamics refines $\tilde{\mathbf{y}}^{(0)}$ using ranking, fluency, and n-gram constraints (§\ref{['subsec:losses']}). (4) The final SRP is then inserted into the target product, elevating its rank without overt trigger words.
• Figure 3: Human evaluation results comparing StealthRank (Ours) and STS. Higher is better for fluency and persuasiveness; lower is better for manipulation detectability. Values in bold indicate preferences for our method. Ours outperforms in all three metrics.