Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers

Yixuan Du, Chenxiao Yu, Haoyan Xu, Ziyi Wang, Yue Zhao, Xiyang Hu

TL;DR

This paper reveals a previously overlooked vulnerability in vision-language model–based product ranking: coordinated adversarial perturbations across both image and text can substantially boost a target item’s ranking. It introduces Multimodal Generative Engine Optimization (MGEO), a white-box framework that jointly optimizes an imperceptible image perturbation $\delta_I$ and a fluent textual suffix $\delta_T$ via alternating gradient-based updates, under stealth constraints. MGEO combines soft-embedding text optimization with constrained PGD image perturbations to exploit cross-modal coupling in the ranking objective, outperforming text-only, image-only, and heuristic baselines in real-world-like experiments. The results demonstrate a synergistic effect from multimodal manipulation and call for robustness, auditing, and defense mechanisms to secure multimodal ranking systems in practice, while noting limitations such as evaluation on a single VLM and static listings.

Abstract

Vision-Language Models (VLMs) are rapidly replacing unimodal encoders in modern retrieval and recommendation systems. While their capabilities are well-documented, their robustness against adversarial manipulation in competitive ranking scenarios remains largely unexplored. In this paper, we uncover a critical vulnerability in VLM-based product search: multimodal ranking attacks. We present Multimodal Generative Engine Optimization (MGEO), a novel adversarial framework that enables a malicious actor to unfairly promote a target product by jointly optimizing imperceptible image perturbations and fluent textual suffixes. Unlike existing attacks that treat modalities in isolation, MGEO employs an alternating gradient-based optimization strategy to exploit the deep cross-modal coupling within the VLM. Extensive experiments on real-world datasets using state-of-the-art models demonstrate that our coordinated attack significantly outperforms text-only and image-only baselines. These findings reveal that multimodal synergy, typically a strength of VLMs, can be weaponized to compromise the integrity of search rankings without triggering conventional content filters.

Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers

TL;DR

This paper reveals a previously overlooked vulnerability in vision-language model–based product ranking: coordinated adversarial perturbations across both image and text can substantially boost a target item’s ranking. It introduces Multimodal Generative Engine Optimization (MGEO), a white-box framework that jointly optimizes an imperceptible image perturbation and a fluent textual suffix via alternating gradient-based updates, under stealth constraints. MGEO combines soft-embedding text optimization with constrained PGD image perturbations to exploit cross-modal coupling in the ranking objective, outperforming text-only, image-only, and heuristic baselines in real-world-like experiments. The results demonstrate a synergistic effect from multimodal manipulation and call for robustness, auditing, and defense mechanisms to secure multimodal ranking systems in practice, while noting limitations such as evaluation on a single VLM and static listings.

Abstract

Vision-Language Models (VLMs) are rapidly replacing unimodal encoders in modern retrieval and recommendation systems. While their capabilities are well-documented, their robustness against adversarial manipulation in competitive ranking scenarios remains largely unexplored. In this paper, we uncover a critical vulnerability in VLM-based product search: multimodal ranking attacks. We present Multimodal Generative Engine Optimization (MGEO), a novel adversarial framework that enables a malicious actor to unfairly promote a target product by jointly optimizing imperceptible image perturbations and fluent textual suffixes. Unlike existing attacks that treat modalities in isolation, MGEO employs an alternating gradient-based optimization strategy to exploit the deep cross-modal coupling within the VLM. Extensive experiments on real-world datasets using state-of-the-art models demonstrate that our coordinated attack significantly outperforms text-only and image-only baselines. These findings reveal that multimodal synergy, typically a strength of VLMs, can be weaponized to compromise the integrity of search rankings without triggering conventional content filters.
Paper Structure (25 sections, 9 equations, 5 figures, 3 tables)

This paper contains 25 sections, 9 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adversarial Attacks on LLM-based Ranking
  4. Adversarial Robustness of Vision-Language Models
  5. Methodology
  6. Problem Formulation
  7. VLM-Based Product Ranking System
  8. Adversarial Threat Model
  9. Adversarial Text Generation
  10. Adversarial Image Generation
  11. PGD-Based Optimization
  12. Multi-Objective Loss Function
  13. Joint Multimodal Attack
  14. Experiments and Discussion
  15. Experimental Setup
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Overview of multimodal ranking attack. A malicious seller jointly optimizes subtle, imperceptible perturbations in both the product image and text description to manipulate the VLM's relevance scoring and elevate the target item's rank.
  • Figure 2: Detailed architecture of Multimodal Generative Engine Optimization (MGEO). The attacker jointly optimizes an adversarial text suffix and an image perturbation to promote the target item in a VLM-based ranking system. The text branch performs gradient-based soft prompt optimization under fluency and n-gram constraints, while the image branch applies PGD under smoothness and magnitude constraints. The two modalities are optimized in an alternating manner to exploit cross-modal interactions within the VLM.
  • Figure 3: Qualitative comparison of text and image generation between our method and the baseline. In this example, our method results in a rank change from $3/10 \to 1/10$ while the baseline method has no rank change at all. (a) Text generated by our method. (b) Text generated by the GPT-4o-mini baseline. (c) Image generated by our method. (d) Image generated by the GPT-Image-1-mini baseline.
  • Figure 4: Visualization of adversarial images under different image-side regularization settings. (a) Original image. (b) Strong regularization $(s{=}10, m{=}10)$, rank change from $8/10 \to 6/10$. (c) Moderate regularization $(s{=}5, m{=}5)$, rank change from $8/10 \to 6/10$. (d) Smoothness removed $(s{=}0, m{=}5)$, rank change from $8/10 \to 6/10$. (e) Magnitude removed $(s{=}5, m{=}0)$, rank change from $8/10 \to 6/10$. (f) No regularization $(s{=}0, m{=}0)$, rank change from $8/10 \to 1/10$. Reducing regularization strength improves attack effectiveness but introduces increasingly severe visual artifacts.
  • Figure 5: A failure case of our attack. (a) Original product image. (b) Adversarial image that successfully promotes the target product from $10/10$ to $1/10$, but introduces visually conspicuous artifacts.