Privacy Protection Against Personalized Text-to-Image Synthesis via Cross-image Consistency Constraints

TL;DR

The paper addresses privacy risks from personalized text-to-image diffusion models by proposing Cross-image Anti-Personalization (CAP). CAP introduces a Gram-matrix based style-consistency loss across perturbed images and a dynamic ratio strategy that adaptively balances reconstruction and consistency losses, resulting in stronger protection than per-image perturbations alone. The method is integrated with existing anti-personalization baselines (Anti-DreamBooth and SimAC) and evaluated on CelebA-HQ and VGGFace2, showing improvements in privacy metrics such as $\text{FDFR}$ and $\text{ISM}$ while maintaining perceptual quality (SER-FIQ, CLIP-IQA). Overall, CAP reframes anti-personalization as a cross-image privacy problem, delivering more robust defenses against impersonation without excessive visual distortion, and it can be extended to broader settings like video or multi-concept personalization. The key contributions are the cross-image framing, the style-based consistency loss, and the dynamic ratio optimization, all validated through extensive experiments.

Abstract

The rapid advancement of diffusion models and personalization techniques has made it possible to recreate individual portraits from just a few publicly available images. While such capabilities empower various creative applications, they also introduce serious privacy concerns, as adversaries can exploit them to generate highly realistic impersonations. To counter these threats, anti-personalization methods have been proposed, which add adversarial perturbations to published images to disrupt the training of personalization models. However, existing approaches largely overlook the intrinsic multi-image nature of personalization and instead adopt a naive strategy of applying perturbations independently, as commonly done in single-image settings. This neglects the opportunity to leverage inter-image relationships for stronger privacy protection. Therefore, we advocate for a group-level perspective on privacy protection against personalization. Specifically, we introduce Cross-image Anti-Personalization (CAP), a novel framework that enhances resistance to personalization by enforcing style consistency across perturbed images. Furthermore, we develop a dynamic ratio adjustment strategy that adaptively balances the impact of the consistency loss throughout the attack iterations. Extensive experiments on the classical CelebHQ and VGGFace2 benchmarks show that CAP substantially improves existing methods.

Figures (4)

• Figure 1: Comparison between Anti-DreamBooth before and after adding our CAP method. Our method further boosts its ability to de-identity. The core idea is to encourage the cross-image consistency when generating protected images.
• Figure 2: Motivation. The first row presents the original input images. The second and third rows display perturbed images with consistent and inconsistent disruption marks, respectively. The fourth and fifth rows show images modified to share the same style or different styles. The first four columns are input images used for personalization, and the last four are samples generated using the learned identity concept.
• Figure 3: Visualization. Here, we present the performance of Anti-DB and SimAC, along with the additional improvements achieved by integrating our CAP method.
• Figure 4: Privacy Protection effect under different perturbation strength.