Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PROMPTMINER: Black-Box Prompt Stealing against Text-to-Image Generative Models via Reinforcement Learning and Fuzz Optimization

Mingzhe Li, Renhao Zhang, Zhiyang Wen, Siqi Pan, Bruno Castro da Silva, Juan Zhai, Shiqing Ma

TL;DR

PROMPTMINER addresses the challenge of recovering text prompts from target images in black-box T2I models by decoupling the task into a reinforcement-learning-based subject inversion and a fuzz-testing-driven modifier optimization. The two-stage design achieves high semantic fidelity and stylistic richness without gradient access or large labeled data, outperforming baselines on multiple datasets and backbones and generalizing to in-the-wild images. The paper also analyzes defense strategies, demonstrating robustness of the approach and providing theoretical justification via potential-based reward shaping to accelerate training. Overall, PromptMiner advances the capability to reconstruct prompts with strong image-text alignment, while highlighting the need for stronger prompt-protection mechanisms in diffusion-based generative systems.

Abstract

Text-to-image (T2I) generative models such as Stable Diffusion and FLUX can synthesize realistic, high-quality images directly from textual prompts. The resulting image quality depends critically on well-crafted prompts that specify both subjects and stylistic modifiers, which have become valuable digital assets. However, the rising value and ubiquity of high-quality prompts expose them to security and intellectual-property risks. One key threat is the prompt stealing attack, i.e., the task of recovering the textual prompt that generated a given image. Prompt stealing enables unauthorized extraction and reuse of carefully engineered prompts, yet it can also support beneficial applications such as data attribution, model provenance analysis, and watermarking validation. Existing approaches often assume white-box gradient access, require large-scale labeled datasets for supervised training, or rely solely on captioning without explicit optimization, limiting their practicality and adaptability. To address these challenges, we propose PROMPTMINER, a black-box prompt stealing framework that decouples the task into two phases: (1) a reinforcement learning-based optimization phase to reconstruct the primary subject, and (2) a fuzzing-driven search phase to recover stylistic modifiers. Experiments across multiple datasets and diffusion backbones demonstrate that PROMPTMINER achieves superior results, with CLIP similarity up to 0.958 and textual alignment with SBERT up to 0.751, surpassing all baselines. Even when applied to in-the-wild images with unknown generators, it outperforms the strongest baseline by 7.5 percent in CLIP similarity, demonstrating better generalization. Finally, PROMPTMINER maintains strong performance under defensive perturbations, highlighting remarkable robustness. Code: https://github.com/aaFrostnova/PromptMiner

PROMPTMINER: Black-Box Prompt Stealing against Text-to-Image Generative Models via Reinforcement Learning and Fuzz Optimization

TL;DR

PROMPTMINER addresses the challenge of recovering text prompts from target images in black-box T2I models by decoupling the task into a reinforcement-learning-based subject inversion and a fuzz-testing-driven modifier optimization. The two-stage design achieves high semantic fidelity and stylistic richness without gradient access or large labeled data, outperforming baselines on multiple datasets and backbones and generalizing to in-the-wild images. The paper also analyzes defense strategies, demonstrating robustness of the approach and providing theoretical justification via potential-based reward shaping to accelerate training. Overall, PromptMiner advances the capability to reconstruct prompts with strong image-text alignment, while highlighting the need for stronger prompt-protection mechanisms in diffusion-based generative systems.

Abstract

Text-to-image (T2I) generative models such as Stable Diffusion and FLUX can synthesize realistic, high-quality images directly from textual prompts. The resulting image quality depends critically on well-crafted prompts that specify both subjects and stylistic modifiers, which have become valuable digital assets. However, the rising value and ubiquity of high-quality prompts expose them to security and intellectual-property risks. One key threat is the prompt stealing attack, i.e., the task of recovering the textual prompt that generated a given image. Prompt stealing enables unauthorized extraction and reuse of carefully engineered prompts, yet it can also support beneficial applications such as data attribution, model provenance analysis, and watermarking validation. Existing approaches often assume white-box gradient access, require large-scale labeled datasets for supervised training, or rely solely on captioning without explicit optimization, limiting their practicality and adaptability. To address these challenges, we propose PROMPTMINER, a black-box prompt stealing framework that decouples the task into two phases: (1) a reinforcement learning-based optimization phase to reconstruct the primary subject, and (2) a fuzzing-driven search phase to recover stylistic modifiers. Experiments across multiple datasets and diffusion backbones demonstrate that PROMPTMINER achieves superior results, with CLIP similarity up to 0.958 and textual alignment with SBERT up to 0.751, surpassing all baselines. Even when applied to in-the-wild images with unknown generators, it outperforms the strongest baseline by 7.5 percent in CLIP similarity, demonstrating better generalization. Finally, PROMPTMINER maintains strong performance under defensive perturbations, highlighting remarkable robustness. Code: https://github.com/aaFrostnova/PromptMiner

Paper Structure

This paper contains 45 sections, 1 theorem, 26 equations, 16 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Threat Model
  5. Overview
  6. RL-Based Prompt Inversion
  7. Fuzz Testing-Powered Prompt Optimization
  8. Evaluation
  9. Experiment Setup
  10. Experiment Result
  11. Ablation Study
  12. User Study
  13. Potential Defenses
  14. Conclusion
  15. More Details of the Used Models
  16. ...and 30 more sections

Key Result

Theorem K.1

Let $V_\pi$, $Q_\pi$ be the original value functions and $V'_\pi$, $Q'_\pi$ those under the shaped reward $r'$. Then for any bounded $\Phi$, the following holds: and therefore implying that $\arg\max_a Q'^*(s,a) = \arg\max_a Q^*(s,a)$, i.e., the optimal policy is preserved.

Figures (16)

  • Figure 1: Illustration of the prompt stealing. Given a target image, the attacker aims to recover the prompt to generate a similar image.
  • Figure 2: Overview of PromptMiner. Our method comprises two phases: (I) a reinforcement learning–based optimization phase to reconstruct the primary subject, and (II) a fuzzing-driven search phase to recover stylistic modifiers.
  • Figure 3: Visualization of generated images compared with target image.
  • Figure 3: Potential defenses against PromptMiner.
  • Figure 4: Impact of query budget on phase I (RL-based Prompt Inversion) and phase II (Fuzz Testing–Powered Prompt Optimization).
  • ...and 11 more figures

Theorems & Definitions (1)

  • Theorem K.1