Camera-LiDAR Cross-modality Gait Recognition

TL;DR

This work tackles cross-modality gait recognition between camera silhouettes and LiDAR point clouds, addressing camera limitations in low light and long-range scenarios as well as LiDAR deployment costs. It introduces CL-Gait, a two-stream network that projects LiDAR data into depth-like representations and aligns the two modalities through a contrastive silhouette-point pre-training (CSPP) objective, supplemented by a large-scale data generation strategy using monocular depth estimation to create paired pseudo data. The approach demonstrates that cross-modality gait recognition is feasible and substantially improves performance over single-modality baselines, achieving an average rank-1 accuracy of 54.21% on the SUSTech1K dataset and showing the effectiveness of depth-based LiDAR input and CSPP. The work also provides insights into the importance of 3D geometry for cross-modality matching and outlines practical implications for deploying gait recognition across varying environmental conditions with reduced data collection costs. Overall, CL-Gait advances cross-modality biometric recognition by bridging camera and LiDAR modalities and offering a scalable pre-training pipeline that mitigates modality gaps.

Abstract

Gait recognition is a crucial biometric identification technique. Camera-based gait recognition has been widely applied in both research and industrial fields. LiDAR-based gait recognition has also begun to evolve most recently, due to the provision of 3D structural information. However, in certain applications, cameras fail to recognize persons, such as in low-light environments and long-distance recognition scenarios, where LiDARs work well. On the other hand, the deployment cost and complexity of LiDAR systems limit its wider application. Therefore, it is essential to consider cross-modality gait recognition between cameras and LiDARs for a broader range of applications. In this work, we propose the first cross-modality gait recognition framework between Camera and LiDAR, namely CL-Gait. It employs a two-stream network for feature embedding of both modalities. This poses a challenging recognition task due to the inherent matching between 3D and 2D data, exhibiting significant modality discrepancy. To align the feature spaces of the two modalities, i.e., camera silhouettes and LiDAR points, we propose a contrastive pre-training strategy to mitigate modality discrepancy. To make up for the absence of paired camera-LiDAR data for pre-training, we also introduce a strategy for generating data on a large scale. This strategy utilizes monocular depth estimated from single RGB images and virtual cameras to generate pseudo point clouds for contrastive pre-training. Extensive experiments show that the cross-modality gait recognition is very challenging but still contains potential and feasibility with our proposed model and pre-training strategy. To the best of our knowledge, this is the first work to address cross-modality gait recognition.

Figures (11)

• Figure 1: Overview of single-modality and cross-modality gait recognition. Single-modality gait recognition takes data of one modality as input and searches within a gallery of the same data type. In contrast, cross-modality gait recognition processes two modalities and identify individuals within a gallery of a different modality.
• Figure 2: The cross-modality network of our proposed CL-Gait. It employs a two-stream architecture that encodes sequences from two modalities into a consistent feature space. HP stands for horizontal pooling, and TP represents temporal pooling.
• Figure 3: The contrastive silhouette-point pre-training (CSPP) approach of CL-Gait. Paired silhouettes and point clouds are taken as inputs, and the backbone of the cross-modality network is pre-trained with contrastive learning loss to align the feature spaces of the two modalities. The pre-training process does not require identity labels of the samples for supervision.
• Figure 4: Illustration of paired gait data generation from single RGB images for contrastive pre-training. The quality of the synthesized data is comparable to the real-world data, making it possible to synthesize large-scale pre-training data.
• Figure 5: Comparison of different input forms of LiDAR data. The results show that the projected and interpolated depth from point cloud works best for the cross-modality matching. This indicates that the 3D geometry information are essential. For each input form, the best-performing model is utilized.