HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpainting

TL;DR

The paper addresses the persistent problem of malformed hands in diffusion-generated images by introducing HandRefiner, a post-processing pipeline that uses a hand mesh–derived depth map to condition inpainting via ControlNet. By discovering a phase transition in the ControlNet guidance, the method enables effective training on synthetic data while maintaining realistic textures, and it can be integrated with existing diffusion models without retraining the base model. Across extensive experiments on datasets like HAGRID and FreiHAND, HandRefiner achieves significant improvements in hand realism, pose accuracy, and detection confidence, as evidenced by objective metrics and human judgments. The work offers practical strategies for fixed or adaptive control strength and demonstrates generalizability to other control signals, with code released for public use.

Abstract

Diffusion models have achieved remarkable success in generating realistic images but suffer from generating accurate human hands, such as incorrect finger counts or irregular shapes. This difficulty arises from the complex task of learning the physical structure and pose of hands from training images, which involves extensive deformations and occlusions. For correct hand generation, our paper introduces a lightweight post-processing solution called $\textbf{HandRefiner}$. HandRefiner employs a conditional inpainting approach to rectify malformed hands while leaving other parts of the image untouched. We leverage the hand mesh reconstruction model that consistently adheres to the correct number of fingers and hand shape, while also being capable of fitting the desired hand pose in the generated image. Given a generated failed image due to malformed hands, we utilize ControlNet modules to re-inject such correct hand information. Additionally, we uncover a phase transition phenomenon within ControlNet as we vary the control strength. It enables us to take advantage of more readily available synthetic data without suffering from the domain gap between realistic and synthetic hands. Experiments demonstrate that HandRefiner can significantly improve the generation quality quantitatively and qualitatively. The code is available at https://github.com/wenquanlu/HandRefiner .

Figures (15)

• Figure 1: Stable Diffusion stablediffusion (first two rows), SDXL sdxl (third row) and Midjourney midjourney (last row) generate malformed hands (left in each pair), e.g., incorrect number of fingers or irregular shapes, which can be effectively rectified by our HandRefiner (right in each pair).
• Figure 2: Overview of HandRefiner. During inference, the depth map fitted by a hand mesh reconstruction model is injected into the inpainting inference processes through ControlNet controlnet. During fine-tuning, synthetic data is utilised to train the depth-based ControlNet to reconstruct hands following an inpainting pipeline. HandRefiner is simple yet effective, greatly enhancing the quality of hand generation.
• Figure 3: Hands generated by using the pre-trained Depth ControlNet in our HandRefiner pipeline.
• Figure 4: Examples of hands in the RHD dataset zb2017hand, which lack real textures.
• Figure 5: Illustration of Phase Transition. The hand shape and pose change to match the depth map when applied control strength is small, while the hand wrinkles and textures disappear at high control strength.