Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence

TL;DR

This paper tackles the challenge of extracting meaningful per-pixel descriptors from diffusion models, where useful representations are distributed across both layers and diffusion timesteps. It introduces Diffusion Hyperfeatures, an interpretable aggregation framework that fuses all intermediate diffusion maps into a compact descriptor via learned mixing weights and bottleneck projections. The method uses inversion features for real images and generation features for synthetic images, achieving state-of-the-art semantic keypoint matching on SPair-71k and demonstrating robust transfer to unseen synthetic data. These findings highlight the diffusion model's rich, spatially and temporally distributed representations and open avenues for synthetic data generation with pseudo-ground-truth keypoints, as well as practical downstream tasks like dense warping.

Abstract

Diffusion models have been shown to be capable of generating high-quality images, suggesting that they could contain meaningful internal representations. Unfortunately, the feature maps that encode a diffusion model's internal information are spread not only over layers of the network, but also over diffusion timesteps, making it challenging to extract useful descriptors. We propose Diffusion Hyperfeatures, a framework for consolidating multi-scale and multi-timestep feature maps into per-pixel feature descriptors that can be used for downstream tasks. These descriptors can be extracted for both synthetic and real images using the generation and inversion processes. We evaluate the utility of our Diffusion Hyperfeatures on the task of semantic keypoint correspondence: our method achieves superior performance on the SPair-71k real image benchmark. We also demonstrate that our method is flexible and transferable: our feature aggregation network trained on the inversion features of real image pairs can be used on the generation features of synthetic image pairs with unseen objects and compositions. Our code is available at https://diffusion-hyperfeatures.github.io.

Figures (14)

• Figure 1: Unlike prior work that hand-selects a subset of raw diffusion features, we extract all feature maps from the diffusion process, varying across both timesteps and layers, and use a lightweight aggregation network to consolidate them into Diffusion Hyperfeatures. For real images, we extract these features from the inversion process, and for synthetic images we extract these features from the generation process. Given a pair of images, we find semantic correspondences by performing a nearest-neighbor search over their Diffusion Hyperfeatures.
• Figure 2: We show an example pair of synthetic images for the prompt "cat sitting in a living room" and the PCA of the features from Layers 4, 10 during both an early and late generation step. While different layers capture different image characteristics (here Layer 4 delineates the face vs. body and Layer 10 captures the edges), these features also evolve and become more fine-grained over time.
• Figure 3: We show an example pair of real images from SPair-71k and the PCA of the features from Layers 4, 10 when extracted at the middle timestep $t=25$. While prior work extracts generation features by noising and denoising the image independently at the specific timestep (left), in our approach we extract inversion features from one continuous chain (right). Extracting features from the same timestep of the inversion chain can produce features more true to original image content.
• Figure 4: Example images from SPair-71k and CUB, the ground-truth user-annotated correspondences, and predicted correspondences from each method.
• Figure 5: The learned mixing weights when aggregating SDv1-5 vs. SDv2-1 features across multiple layers and timesteps. Bright yellow denotes a high weighting, and dark blue denotes a low weighting. We also depict predicted correspondences from SDv2-1-Layer-4 vs. Ours-SDv2-1. While Layer 4 features from SDv1-5 perform well in semantic correspondence, this same layer in SDv2-1 performs extremely poorly. Our method automatically learns the best layers depending on the model variant.