Panoramic Distortion-Aware Tokenization for Person Detection and Localization in Overhead Fisheye Images

TL;DR

This work tackles the challenge of detecting and localizing people in overhead fisheye images, where rotation and small object size hinder performance. It remaps fisheye frames to equirectangular panoramas and introduces panoramic distortion-aware tokenization (PDAT), which uses a self-similar tiling to balance significance across regions and preserve small-person cues during transformer-based detection. The method fuses a DAB-DETR detector with an SG-Former backbone and integrates PDAT to maintain small-object significance, achieving state-of-the-art results across LOAF, CEPDOF, and WEPDTOF datasets and improving localization accuracy. The findings suggest that geometry-guided panorama tiling and distortion-aware tokenization substantially enhance detection and localization in wide-field overhead imagery, with practical impact for surveillance and analytics in large-scale setups.

Abstract

Person detection in overhead fisheye images is challenging due to person rotation and small persons. Prior work has mainly addressed person rotation, leaving the small-person problem underexplored. We remap fisheye images to equirectangular panoramas to handle rotation and exploit panoramic geometry to handle small persons more effectively. Conventional detection methods tend to favor larger persons because they dominate the attention maps, causing smaller persons to be missed. In hemispherical equirectangular panoramas, we find that apparent person height decreases approximately linearly with the vertical angle near the top of the image. Using this finding, we introduce panoramic distortion-aware tokenization to enhance the detection of small persons. This tokenization procedure divides panoramic features using self-similar figures that enable the determination of optimal divisions without gaps, and we leverage the maximum significance values in each tile of the token groups to preserve the significance areas of smaller persons. We propose a transformer-based person detection and localization method that combines panoramic-image remapping and the tokenization procedure. Extensive experiments demonstrated that our method outperforms conventional methods on large-scale datasets.

Figures (4)

• Figure 1: Our method converts an overhead fisheye image into an equirectangular image by remapping. To leverage the significance areas of smaller persons, a distortion-aware tokenization process based on a self-similar figure using a unit figure, which is colored dark red and consists of a square and two rectangles. Our transformer-based detector uses the resulting tokens to detect bounding boxes within the panoramic image, and then the boxes project to the fisheye image. The input image is taken from Yang_ICCV_2023.
• Figure 2: Analysis of overhead fisheye images. (a) An equirectangular image. The vertical and horizontal axes show the incident angle (0$^\circ$ is at the bottom of the image) and the azimuth angle, respectively. (b) Distribution of the bounding boxes in equirectangular images on the LOAF training set. This distribution was analyzed using incident angles $\theta$ at 1$^\circ$-intervals based on the center of the bounding boxes. The top and middle panels represent the mean heights and widths of the bounding boxes in equirectangular images with 3072$\times$768 pixels, respectively. The cyan areas indicate $\pm1$ standard deviations in each bin of incident angles. The bottom panel shows the number of bounding boxes.
• Figure 3: Panoramic distortion-aware tokenization. (a) A panoramic feature map is nonuniformly tiled with cyan grids along the incident-angle axis. The tile sizes are adjusted to match the person's height in each region, maintaining similar person-to-tile size ratios across different regions. Enlarged views show person-bounding boxes in regions 3, 4, and 5. (b) A self-similar subdivision recursively partitions the unit figure and is truncated after three steps, yielding five regions ($K=5$) with the size ratios of the unit figure: 1, 1/2, 1/4, and 1/8.
• Figure 4: Overall network architecture of our proposed method. The model consists of a detector head and a backbone with four stages. The $N$ in the subscript stage indicates the block count. The $\mathbf{S}$ and $\mathbf{X}$ are the significance map and feature map, respectively.