$CrowdDiff$: Multi-hypothesis Crowd Density Estimation using Diffusion Models

TL;DR

CrowdDiff reframes crowd density estimation as a conditional diffusion process, addressing background noise and density loss from broad kernels by using a narrow Gaussian kernel and a diffusion-based density map predictor. A counting branch trained with intermediary diffusion features and a multi-realization crowd map fusion strategy leverages the stochastic nature of diffusion to boost counting accuracy. The method achieves state-of-the-art performance on major crowd datasets, with ablations showing the benefits of the counting branch, careful fusion order, and kernel choices. By thresholding density maps rather than summing pixel values, CrowdDiff attains robust counts in dense scenes and eliminates the need for density-based heuristics, offering a scalable, multi-hypothesis counting framework for real-world surveillance scenarios.

Abstract

Crowd counting is a fundamental problem in crowd analysis which is typically accomplished by estimating a crowd density map and summing over the density values. However, this approach suffers from background noise accumulation and loss of density due to the use of broad Gaussian kernels to create the ground truth density maps. This issue can be overcome by narrowing the Gaussian kernel. However, existing approaches perform poorly when trained with ground truth density maps with broad kernels. To deal with this limitation, we propose using conditional diffusion models to predict density maps, as diffusion models show high fidelity to training data during generation. With that, we present $CrowdDiff$ that generates the crowd density map as a reverse diffusion process. Furthermore, as the intermediate time steps of the diffusion process are noisy, we incorporate a regression branch for direct crowd estimation only during training to improve the feature learning. In addition, owing to the stochastic nature of the diffusion model, we introduce producing multiple density maps to improve the counting performance contrary to the existing crowd counting pipelines. We conduct extensive experiments on publicly available datasets to validate the effectiveness of our method. $CrowdDiff$ outperforms existing state-of-the-art crowd counting methods on several public crowd analysis benchmarks with significant improvements.

Figures (11)

• Figure 1: Predicted density results for (a) a dense crowd from (b) our method, (c) Chfl shu2022crowd, and (d) SUA meng2021spatial. The count of the enlarged crop is given in brackets.
• Figure 2: Overall crowd counting pipeline. The crowd density maps are generated from the denoising diffusion process for a crowd image. Next, thresholding is performed on the resulting crowd density realizations to create crowd maps. The crowd maps are then fused into a single crowd map. The counting branch is trained in parallel using the encoder-decoder features of the denoising U-Net and discarded during inference.
• Figure 3: Change in the pixel values of the Gaussian kernel (red stems) and the resulting density map (blue stems) for a crowd image with a $3,547$ crowd count. The kernel size and variance increase from left to right.
• Figure 4: Crowd map fusion criterion. The rejection radius is computed from the neighbors (black) inside the neighbor radius. New points (colored) that fall inside the rejection radius are removed (red), and the rest (green) are combined into the compound map.
• Figure 5: Qualitative results for the proposed method with the ground truths. The prediction is produced after combining multiple realizations.