Exploring Scale Shift in Crowd Localization under the Context of Domain Generalization

TL;DR

This work investigates how head-scale distribution shifts hinder crowd localization under domain generalization, introducing ScaleBench to quantify scale-shift effects and Catto to mitigate them. ScaleBench uses patch-level, scale-aware domain partitions and reproduces 20 DG methods, revealing that many DG approaches struggle with scale shifts in dense crowds. The authors formalize scale shift as a mixed diversity and correlation domain shift and prove its impact via Theorem SSAD, then propose Catto to disentangle semantic and scale features with semantic hooks, scale-elimination via image interpolation, and invariant learning. Empirical results show Catto achieving state-of-the-art DG performance on ScaleBench and yield four practical insights for future scale-shift research in crowd localization and beyond.

Abstract

Crowd localization plays a crucial role in visual scene understanding towards predicting each pedestrian location in a crowd, thus being applicable to various downstream tasks. However, existing approaches suffer from significant performance degradation due to discrepancies in head scale distributions (scale shift) between training and testing data, a challenge known as domain generalization (DG). This paper aims to comprehend the nature of scale shift within the context of domain generalization for crowd localization models. To this end, we address four critical questions: (i) How does scale shift influence crowd localization in a DG scenario? (ii) How can we quantify this influence? (iii) What causes this influence? (iv) How to mitigate the influence? Initially, we conduct a systematic examination of how crowd localization performance varies with different levels of scale shift. Then, we establish a benchmark, ScaleBench, and reproduce 20 advanced DG algorithms to quantify the influence. Through extensive experiments, we demonstrate the limitations of existing algorithms and underscore the importance and complexity of scale shift, a topic that remains insufficiently explored. To deepen our understanding, we provide a rigorous theoretical analysis on scale shift. Building on these insights, we further propose an effective algorithm called Causal Feature Decomposition and Anisotropic Processing (Catto) to mitigate the influence of scale shift in DG settings. Later, we also provide extensive analytical experiments, revealing four significant insights for future research. Our results emphasize the importance of this novel and applicable research direction, which we term Scale Shift Domain Generalization.

Figures (16)

• Figure 1: Task Formulations of (a) Crowd Localization and (b) Domain Generalization with Scale Shift. In (b), the primary difference among domains is the average scale of objects. In real-world applications, the number of source domains can vary. In our subsequent setup, the division between source and target domains is also flexible.
• Figure 2: Curve illustrating the variation in model performance with respect to scale shift. This graph depicts the relationship between the average object scale in the testing domains and the corresponding performance metrics of the trained crowd locator, specifically highlighting how performance declines with increasing scale shift.
• Figure 3: This figure outlines the systematic pipeline for generating domains in ScaleBench, a benchmark dataset designed to study domain generalization under scale shifts for crowd localization tasks. The process begins with regularizing the image-level scale distribution by filtering out unqualified samples based on predefined criteria, such as patch height thresholds and standard deviation constraints. Following regularization, the overall scale distribution is analyzed statistically, and patches are represented using their mean scale. Based on this analysis, the patches are systematically divided into four distinct domains—Tiny, Small, Normal, and Big—to capture inter-domain scale shifts while maintaining intra-domain consistency. Probability density functions (PDFs) for each domain highlight the scale characteristics and diversity across the dataset. This pipeline ensures that ScaleBench captures diverse scale distributions, enabling comprehensive evaluation of model performance under varying scale shifts and facilitating robust domain generalization in crowd analysis tasks.
• Figure 4: Training pipeline of our proposed Catto.
• Figure 5: Left is the statistic for the number of image and pedestrian within each dataset, where the number of pedestrian we annotated with boxes are in noted in red; Mid is the statistic for the resolution distributions; Right is the boxplot for the scale distribution, where the star denotes the mean value, the dash line within the box denotes the median value.

Theorems & Definitions (12)

• theorem 1: Scale Shift as A Mixed Domain Shift
• definition thmcounterdefinition: Scale Distribution
• definition thmcounterdefinition: Random Finite Sets
• lemma thmcounterlemma: Domain Shift OODBench
• proposition thmcounterproposition: Linear Expression Among Object Scales
• definition thmcounterdefinition: Feature Sets
• definition thmcounterdefinition
• remark thmcounterremark
• lemma thmcounterlemma: Definition 1 in OODBench
• lemma thmcounterlemma: Proposition 1 in OODBench