MONKEY: Masking ON KEY-Value Activation Adapter for Personalization
James Baker
TL;DR
Personalization in diffusion models often trades off subject fidelity against prompt alignment. The authors introduce MONKEY Adapter, a training-free two-pass inference that derives an implicit subject mask from IP-Adapter activations and masks image tokens to isolate the subject, allowing the background to be guided by the text prompt. This approach improves both subject preservation and background/plausible context alignment without additional training, and it sits on the Pareto frontier for text and image alignment across benchmarks. The work demonstrates enhanced controllability for personalized generations and points to future directions such as multi-subject personalization and combination with conditioning methods like ControlNet.
Abstract
Personalizing diffusion models allows users to generate new images that incorporate a given subject, allowing more control than a text prompt. These models often suffer somewhat when they end up just recreating the subject image and ignoring the text prompt. We observe that one popular method for personalization, IP-Adapter, automatically generates masks that segment the subject from the background during inference. We propose to use this automatically generated mask on a second pass to mask the image tokens, thus restricting them to the subject, not the background, allowing the text prompt to attend to the rest of the image. For text prompts describing locations and places, this produces images that accurately depict the subject while definitively matching the prompt. We compare our method to a few other test time personalization methods, and find our method displays high prompt and source image alignment. We also perform a user study to validate whether end users would appreciate our method. Code available at https://github.com/jamesBaker361/monkey