Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ImPoster: Text and Frequency Guidance for Subject Driven Action Personalization using Diffusion Models

Divya Kothandaraman, Kuldeep Kulkarni, Sumit Shekhar, Balaji Vasan Srinivasan, Dinesh Manocha

TL;DR

ImPoster tackles the problem of generating a target image of a source subject performing a driving action specified by a driving image and text, using only a single source-driving pair in an unsupervised setting. The method fine-tunes a pretrained diffusion model on the pair and employs a two-stage inference strategy: first guiding the denoising toward the driving image manifold to encode the action, then toward the target text to encode the subject, reinforced at every step by frequency guidance in the Fourier domain. Frequency guidance combines amplitude and phase cues, with $G_a$ aligning source-subject characteristics and $G_p$ preserving driving-pose, implemented through latent updates $ ilde{z_t} = z_t - s_a abla_{z_t}G_a - s_p abla_{z_t}G_p$. The authors contribute (i) a formal task definition and a diffusion-based solution, (ii) a 120-pair dataset and a new Phase score metric, and (iii) extensive experiments showing gains over baselines and ablations. This work enables controllable, pose-specific personalization and lays groundwork for future extensions to video and multi-subject scenarios while highlighting societal implications of realistic synthetic imagery.

Abstract

We present ImPoster, a novel algorithm for generating a target image of a 'source' subject performing a 'driving' action. The inputs to our algorithm are a single pair of a source image with the subject that we wish to edit and a driving image with a subject of an arbitrary class performing the driving action, along with the text descriptions of the two images. Our approach is completely unsupervised and does not require any access to additional annotations like keypoints or pose. Our approach builds on a pretrained text-to-image latent diffusion model and learns the characteristics of the source and the driving image by finetuning the diffusion model for a small number of iterations. At inference time, ImPoster performs step-wise text prompting i.e. it denoises by first moving in the direction of the image manifold corresponding to the driving image followed by the direction of the image manifold corresponding to the text description of the desired target image. We propose a novel diffusion guidance formulation, image frequency guidance, to steer the generation towards the manifold of the source subject and the driving action at every step of the inference denoising. Our frequency guidance formulations are derived from the frequency domain properties of images. We extensively evaluate ImPoster on a diverse set of source-driving image pairs to demonstrate improvements over baselines. To the best of our knowledge, ImPoster is the first approach towards achieving both subject-driven as well as action-driven image personalization. Code and data is available at https://github.com/divyakraman/ImPosterDiffusion2024.

ImPoster: Text and Frequency Guidance for Subject Driven Action Personalization using Diffusion Models

TL;DR

ImPoster tackles the problem of generating a target image of a source subject performing a driving action specified by a driving image and text, using only a single source-driving pair in an unsupervised setting. The method fine-tunes a pretrained diffusion model on the pair and employs a two-stage inference strategy: first guiding the denoising toward the driving image manifold to encode the action, then toward the target text to encode the subject, reinforced at every step by frequency guidance in the Fourier domain. Frequency guidance combines amplitude and phase cues, with aligning source-subject characteristics and preserving driving-pose, implemented through latent updates . The authors contribute (i) a formal task definition and a diffusion-based solution, (ii) a 120-pair dataset and a new Phase score metric, and (iii) extensive experiments showing gains over baselines and ablations. This work enables controllable, pose-specific personalization and lays groundwork for future extensions to video and multi-subject scenarios while highlighting societal implications of realistic synthetic imagery.

Abstract

We present ImPoster, a novel algorithm for generating a target image of a 'source' subject performing a 'driving' action. The inputs to our algorithm are a single pair of a source image with the subject that we wish to edit and a driving image with a subject of an arbitrary class performing the driving action, along with the text descriptions of the two images. Our approach is completely unsupervised and does not require any access to additional annotations like keypoints or pose. Our approach builds on a pretrained text-to-image latent diffusion model and learns the characteristics of the source and the driving image by finetuning the diffusion model for a small number of iterations. At inference time, ImPoster performs step-wise text prompting i.e. it denoises by first moving in the direction of the image manifold corresponding to the driving image followed by the direction of the image manifold corresponding to the text description of the desired target image. We propose a novel diffusion guidance formulation, image frequency guidance, to steer the generation towards the manifold of the source subject and the driving action at every step of the inference denoising. Our frequency guidance formulations are derived from the frequency domain properties of images. We extensively evaluate ImPoster on a diverse set of source-driving image pairs to demonstrate improvements over baselines. To the best of our knowledge, ImPoster is the first approach towards achieving both subject-driven as well as action-driven image personalization. Code and data is available at https://github.com/divyakraman/ImPosterDiffusion2024.
Paper Structure (18 sections, 3 equations, 10 figures, 1 table)

This paper contains 18 sections, 3 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Main contributions.
  3. Related work
  4. Exemplar image animation.
  5. Diffusion models for Text-based image editing/ personalization.
  6. Guidance methods for diffusion.
  7. Method
  8. Training
  9. Inference
  10. Stepwise text prompting
  11. Frequency guidance
  12. Experiments and Results
  13. Dataset and implementation details.
  14. State-of-the-art comparisons.
  15. Quantitative Results: Comparisons with Baseline++ and Ablations
  16. ...and 3 more sections

Figures (10)

  • Figure 1: Given a single "source" image and a "single" driving image and the corresponding text descriptions, ImPoster generates an image of the source subject performing driving action. We show how ImPoster is able to make a monkey and an alien meditate and play violin.
  • Figure 2: Given a single source-driving image pair, ImPoster generates an image of the source subject performing the action depicted in the driving image. ImPoster first finetunes the text-to-image diffusion model on the source-driving image pair. At inference, ImPoster begins by first denoising in the direction of the driving image manifold followed by moving towards the manifold corresponding to the desired target image. At every step of the inference, frequency guidance steers the generation of an image with source subject characteristics and driving action.
  • Figure 3: ImPoster is able to successfully transfer the driving action to a source subject, while maintaining its characteristics. In contrast, Baseline++ is unable to generate the driving action for the given source subject due to bias issues. Please see the appendix for more results generated using ImPoster and comparisons with Baseline++.
  • Figure 4: ImPoster is able to successfully transfer the driving motion while retaining the characteristics of the source subject, and achieves a better trade-off between driving action (CLIP/Phase score) and source subject (SSCD/DINO) than prior work, as also evidenced by our qualitative results. Stepwise text prompting creates a prior for the driving action to enable the model generate the driving action. The image frequency guidance formulations help the model in preserving the characteristics of the source subject, while reinforcing the driving action.
  • Figure 5: Ablations. Without stepwise text prompting, there is no prior for the driving action, which inhibits the model from generating the driving action accurately. The frequency (amplitude and phase) guidance methods help in generating the characteristics of the source subject (here, monkey) accurately - notice that there are changes to the color of the monkey (column 6), missing details in the fingers (column 4), changes in the size of the monkey (column 5).
  • ...and 5 more figures