Guiding Diffusion with Deep Geometric Moments: Balancing Fidelity and Variation

TL;DR

This work tackles the challenge of achieving fine-grained, subject-specific control in diffusion-based image synthesis without retraining. It introduces Deep Geometric Moments (DGM) as a training-free guidance signal that encodes geometric priors of the target subject via a pretrained DGM encoder and guides sampling with a per-step feature loss, augmented by a noise-reinjection strategy to preserve diversity. Evaluations on the DGMBench dataset show that DGM achieves a favorable balance between fidelity to the reference and generation diversity, outperforming several training-free baselines and offering advantages over training-based approaches that tend to overfit to either global semantics or exact replicas. The proposed method enables flexible, subject-centric control for T2I systems and can serve as a plug-in prior to enhance personalized diffusion models without additional training.

Abstract

Text-to-image generation models have achieved remarkable capabilities in synthesizing images, but often struggle to provide fine-grained control over the output. Existing guidance approaches, such as segmentation maps and depth maps, introduce spatial rigidity that restricts the inherent diversity of diffusion models. In this work, we introduce Deep Geometric Moments (DGM) as a novel form of guidance that encapsulates the subject's visual features and nuances through a learned geometric prior. DGMs focus specifically on the subject itself compared to DINO or CLIP features, which suffer from overemphasis on global image features or semantics. Unlike ResNets, which are sensitive to pixel-wise perturbations, DGMs rely on robust geometric moments. Our experiments demonstrate that DGM effectively balance control and diversity in diffusion-based image generation, allowing a flexible control mechanism for steering the diffusion process.

Figures (8)

• Figure 1: Deep Geometric Moments (DGM) are used as guidance to capture nuanced subject details from the reference image while preserving the generative diversity of the diffusion model. Each generated result is conditioned on the prompt “a photo of < animal name>” For each pair, the left image is the reference, and the right image is the generated result. The small image in the bottom-left corner of the reference is the DGM visualization used for guidance.
• Figure 2: Failure cases of guidance based methods. Spatial rigidity uses segmentation maps as guidance bansal2023universal; Semantic rigidity shows results of CLIP image features as guidance.
• Figure 3: Qualitative Comparison of different methods. The leftmost image in each row is the reference image. Our method, which uses DGM as guidance, is followed by four alternative feature descriptors and two training-based methods. Overall, our DGM-based guidance achieves a balanced trade-off between visual consistency and generative diversity in the diffusion output.
• Figure 4: Failure examples of our method.
• Figure 5: Additional qualitative comparison results for the prompt: "A photo of a bird". None of the results are cherry-picked.