Real-Time Crowd Counting for Embedded Systems with Lightweight Architecture

TL;DR

This work targets real-time crowd counting on embedded systems by introducing a lightweight stem-encoder-decoder architecture. It combines a large-kernel stem network, conditional channel weighting (CCW), and multi-branch local fusion (MLF) in the encoder, with a Feature Pyramid Network (FPN) decoder to achieve fast, energy-efficient inference while maintaining competitive accuracy. Key contributions include (1) a novel architecture optimized for embedded devices, (2) the first application of CCW in crowd counting and the accompanying MLF fusion strategy, and (3) state-of-the-art inference efficiency demonstrated by 381.7 FPS on GTX 1080Ti and 71.9 FPS on Jetson TX1, supported by the Accuracy–Efficiency Score metric. These results enable practical real-time deployment for surveillance and public safety tasks on low-power hardware.

Abstract

Crowd counting is a task of estimating the number of the crowd through images, which is extremely valuable in the fields of intelligent security, urban planning, public safety management, and so on. However, the existing counting methods have some problems in practical application on embedded systems for these fields, such as excessive model parameters, abundant complex calculations, etc. The practical application of embedded systems requires the model to be real-time, which means that the model is fast enough. Considering the aforementioned problems, we design a super real-time model with a stem-encoder-decoder structure for crowd counting tasks, which achieves the fastest inference compared with state-of-the-arts. Firstly, large convolution kernels in the stem network are used to enlarge the receptive field, which effectively extracts detailed head information. Then, in the encoder part, we use conditional channel weighting and multi-branch local fusion block to merge multi-scale features with low computational consumption. This part is crucial to the super real-time performance of the model. Finally, the feature pyramid networks are added to the top of the encoder to alleviate its incomplete fusion problems. Experiments on three benchmarks show that our network is suitable for super real-time crowd counting on embedded systems, ensuring competitive accuracy. At the same time, the proposed network reasoning speed is the fastest. Specifically, the proposed network achieves 381.7 FPS on NVIDIA GTX 1080Ti and 71.9 FPS on NVIDIA Jetson TX1.

Figures (5)

• Figure 1: (a): Comparison with other networks in parameter quantity and FPS. FPS is tested on NVIDIA GTX 1080Ti. (b): This chart shows the performance comparison of different methods in terms of frame rate (FPS) and mean absolute error (MAE) on the SHHA dataset. 'Ours' method achieves the highest frame rate (approximately 380 FPS) and the lowest mean absolute error (approximately 65 MAE), outperforming other existing methods in both speed and accuracy.
• Figure 2: Our network for crowd counting tasks. The left dotted line in the figure is the stem network and encoder structure, which passes through the stem network (containing three convolutions and shuffle blocks) and two stages (containing two CCW blocks and one MLF block respectively). CCW stands for conditional channel weighting and MLF stands for multi-branch local fusion. The right dotted line is the decoder structure, and the output density map is finally obtained through feature pyramid networks (FPN) and two layers for feature regression.
• Figure 3: Two types of blocks. In (b), H in the first orange box represents the cross-resolution weight function from different branches, where the line of dashes means the weight representation assigned to other branches. In the other orange box, F represents the spatial weight function.
• Figure 4: The first row displays images on NWPU-Crowd, ShanghaiTech, and UCF-QNRF test sets. The second row lists the corresponding ground truth density maps (GT). The third row lists predicted density maps (Est). Numbers represent the actual and predicted number of people, respectively.
• Figure 5: Tables which are too long to fit, should be written using the "sidewaystable" environment as shown here