Crowded Video Individual Counting Informed by Social Grouping and Spatial-Temporal Displacement Priors

TL;DR

The paper tackles VIC in densely crowded scenes by rethinking matching as one-to-many (O2M) rather than one-to-one (O2O) and by embedding motion priors into the model. It introduces WuhanMetroCrowd, a large surveillance-based VIC dataset with diverse density, long clips, and heavy occlusions, to address data gaps in crowded scenarios. The proposed OMAN++ framework combines an Implicit Context Generator, a Displacement Prior Injector, and an OT-based loss to fuse social grouping context with spatial-temporal motion cues, achieving strong gains across SenseCrowd, CroHD, MovingDroneCrowd, and especially WuhanMetroCrowd. The results demonstrate the practical impact of leveraging grouping and displacement priors for robust VIC in real-world, crowded environments, with code and models publicly available.

Abstract

Video Individual Counting (VIC) is a recently introduced task aiming to estimate pedestrian flux from a video. It extends Video Crowd Counting (VCC) beyond the per-frame pedestrian count. In contrast to VCC that learns to count pedestrians across frames, VIC must identify co-existent pedestrians between frames, which turns out to be a correspondence problem. Existing VIC approaches, however, can underperform in congested scenes such as metro commuting. To address this, we build WuhanMetroCrowd, one of the first VIC datasets that characterize crowded, dynamic pedestrian flows. It features sparse-to-dense density levels, short-to-long video clips, slow-to-fast flow variations, front-to-back appearance changes, and light-to-heavy occlusions. To better adapt VIC approaches to crowds, we rethink the nature of VIC and recognize two informative priors: i) the social grouping prior that indicates pedestrians tend to gather in groups and ii) the spatial-temporal displacement prior that informs an individual cannot teleport physically. The former inspires us to relax the standard one-to-one (O2O) matching used by VIC to one-to-many (O2M) matching, implemented by an implicit context generator and a O2M matcher; the latter facilitates the design of a displacement prior injector, which strengthens not only O2M matching but also feature extraction and model training. These designs jointly form a novel and strong VIC baseline OMAN++. Extensive experiments show that OMAN++ not only outperforms state-of-the-art VIC baselines on the standard SenseCrowd, CroHD, and MovingDroneCrowd benchmarks, but also indicates a clear advantage in crowded scenes, with a 38.12% error reduction on our WuhanMetroCrowd dataset. Code, data, and pretrained models are available at https://github.com/tiny-smart/OMAN.

Figures (11)

• Figure 1: Video individual counting in crowded scenes. (a) The key of VIC is to identify co-existent pedestrians between frames, but (b) existing methods typically apply a O2O matching strategy, suffering from missing detections due to occlusions. (c) Inspired by the grouping prior of walking pedestrians, we relax the O2O matching to O2M matching that allows an individual to match a group to enhance matching robustness. In addition, (d) matching informed by only appearance cues may not distinguish visually similar pedestrians, leading to wrong matches. In fact, (e) walking pedestrians obey spatial-temporal displacement prior that encourage spatially close matches.
• Figure 2: Two priors behind moving crowds. (a) The social grouping prior indicating that pedestrians tend to walk in groups, with each social group connected with lines of different colors. (b) The spatial-temporal displacement prior revealing a limited range of pedestrian displacements (left) and their min-max normalized displacements (right) within three-second intervals, suggesting pedestrian motion is physically bounded.
• Figure 3: Overview of WuhanMetroCrowd dataset. This map shows the geographical distribution of data collection, and annotation number of each station.
• Figure 4: Annotated samples from WuhanMetroCrowd dataset. Scenes are shown on the top, and some relatively sparse scenes are shown in the first row while dense scenes in the second row. Green, red, light blue, and yellow points respectively represent the class 'pedestrian', 'inflow', 'outflow', and 'both', and transparent dark blue polygon regions represent masks. 'S', 'N', 'C', 'P', and 'J' on the left-top of each picture, separately represents $5$ different density levels, 'Sparse', 'Normal', 'Crowded', 'Packed', and 'Jam'.
• Figure 5: Density and variation distribution of WuhanMetroCrowd dataset. (a) depicts the statistical overview of our datasets. Each point denotes a video clip, X-axis represents frame number it contains, Y-axis counts its total annotation number, its scale represents the average density across the video, and the color denotes its belonging scene. (b) shows the dataset partition w.r.t scenes. (c) split class 'inflow', 'outflow', 'pedestrian', and total counts into five density levels. (d) similarly split them into five variation levels, illustrating that a single scene of WuhanMetroCrowd dataset may cover both static and dynamic situation.