Key-point Guided Deformable Image Manipulation Using Diffusion Model

TL;DR

A Key-point-guided Diffusion probabilistic Model (KDM) that gains precise control over images by manipulating the object's key-point is introduced, and a two-stage generative model incorporating an optical flow map as an intermediate output is proposed.

Abstract

In this paper, we introduce a Key-point-guided Diffusion probabilistic Model (KDM) that gains precise control over images by manipulating the object's key-point. We propose a two-stage generative model incorporating an optical flow map as an intermediate output. By doing so, a dense pixel-wise understanding of the semantic relation between the image and sparse key point is configured, leading to more realistic image generation. Additionally, the integration of optical flow helps regulate the inter-frame variance of sequential images, demonstrating an authentic sequential image generation. The KDM is evaluated with diverse key-point conditioned image synthesis tasks, including facial image generation, human pose synthesis, and echocardiography video prediction, demonstrating the KDM is proving consistency enhanced and photo-realistic images compared with state-of-the-art models.

Figures (14)

• Figure 1: Illustration of the proposed KDM framework is presented. The source image and corresponding key-point of the facial expression generation, human pose synthesis, and echocardiography video synthesis are introduced.
• Figure 2: Overview of the image manipulation process.
• Figure 3: Qualitative studies of the facial landmark guided image synthesis.
• Figure 4: Qualitative result on the continuous facial image generation. In order to visualize the optical flow fields, the flow field is color-coded as proposed in butler2012naturalistic.
• Figure 5: Qualitative study of the key-point guided human pose synthesis.