Negative Token Merging: Image-based Adversarial Feature Guidance

TL;DR

The paper tackles the limitation of text-only adversarial guidance in diffusion models by introducing NegToMe, a training-free, image-based approach that guides generation through cross-image visual features during reverse diffusion. It leverages cross-token semantic matching between generated outputs and reference images to push source tokens away from their best-matching targets, using the update $\mathrm{O}_{merge} = (1 + \alpha_t) \mathrm{O}_{src} - \alpha_t \mathrm{O}_{target}$ with a threshold $\tau$, and can be extended to MM-DiT architectures like Flux. Empirically, NegToMe improves output diversity (lower DreamSim, higher Entropy) and reduces resemblance to copyrighted content (e.g., $34.57\%$ reduction) with <4\% extra inference time, while maintaining or improving standard quality metrics (FID, IS, CLIPScore, VQAScore). The method is simple to implement, model-agnostic, and adaptable (including masking and time-varying alpha), enabling diverse applications from diversity control to copyright mitigation in diffusion-based generation workflows.

Abstract

Text-based adversarial guidance using a negative prompt has emerged as a widely adopted approach to steer diffusion models away from producing undesired concepts. While useful, performing adversarial guidance using text alone can be insufficient to capture complex visual concepts or avoid specific visual elements like copyrighted characters. In this paper, for the first time we explore an alternate modality in this direction by performing adversarial guidance directly using visual features from a reference image or other images in a batch. We introduce negative token merging (NegToMe), a simple but effective training-free approach which performs adversarial guidance through images by selectively pushing apart matching visual features between reference and generated images during the reverse diffusion process. By simply adjusting the used reference, NegToMe enables a diverse range of applications. Notably, when using other images in same batch as reference, we find that NegToMe significantly enhances output diversity (e.g., racial, gender, visual) by guiding features of each image away from others. Similarly, when used w.r.t. copyrighted reference images, NegToMe reduces visual similarity to copyrighted content by 34.57%. NegToMe is simple to implement using just few-lines of code, uses only marginally higher (<4%) inference time and is compatible with different diffusion architectures, including those like Flux, which don't natively support the use of a negative prompt. Code is available at https://negtome.github.io

Figures (12)

• Figure 1: Image-based adversarial guidance. NegToMe enables directly using images (instead of negative prompt alone) for adversarial guidance. By simply adjusting the used reference, NegToMe allows for a range of custom applications, 1) adversarial guidance for visually complex concepts to improve diversity, 2) Style control for excluding specific artistic styles (e.g., Van Gogh) from generated images, 3) improving output quality by guiding away from a blurry reference, 4) Object feature interpolation or extrapolation by guiding the outputs towards or away from the reference etc.
• Figure 2: Method Overview. (a) The core idea of NegToMe is to perform adversarial guidance directly using visual features from a reference image (or other images in the same batch). (b) NegToMe is simple and can be applied in any transformer block. (c) A simple three step process for performing adversarial guidance using NegToMe (refer Sec. \ref{['sec:method']} and Alg. \ref{['alg:negtome']} for the detailed implementation).
• Figure 3: Quantitative Results for Output Diversity. NegToMe (ours) helps improve output diversity (lower DreamSim score and higher Entropy) while preserving or improving quality (lower FID and higher IS) across different CFG scales for both SDXL and FLUX.
• Figure 4: Increasing Output Diversity. We observe that when performed w.r.t to images in the same batch (the first image of each batch in above), NegToMe significantly improves output diversity (racial, ethnic, visual) while preserving output image quality.
• Figure 5: NegToMe helps improve output diversity both with (row-2) and without explicit prompt-rewriting (PW) (row-4).