Person Recognition at Altitude and Range: Fusion of Face, Body Shape and Gait

TL;DR

FarSight tackles robust whole-body biometric recognition at long standoff distances and elevated viewpoints by fusing face, gait, and body-shape cues through an end-to-end four-module pipeline. It introduces turbulence-aware video restoration (GRTM) co-optimized with recognition, and upgrades modality encoders with KP-RPE for faces, BigGait for gait, and CLIP3DReID for body shape, plus a quality-guided fusion framework (QE and QME) to adapt to input quality. Extensive BRIAR EVP 5.0.1 and NIST FIVE evaluations show substantial gains over the prior system, including a $34.1\%$ absolute improvement in 1:1 TAR, $17.8\%$ in Rank-20, and $34.3\%$ reduction in FNIR@1\% FPIR, with competitive public benchmarks. A publicly trained FarSight variant demonstrates reproducibility and practical applicability, while system optimizations ensure scalable, real-time deployment on multi-GPU hardware. Overall, FarSight establishes a state-of-the-art, open-set capable solution for operational biometric recognition in challenging real-world conditions.

Abstract

We address the problem of whole-body person recognition in unconstrained environments. This problem arises in surveillance scenarios such as those in the IARPA Biometric Recognition and Identification at Altitude and Range (BRIAR) program, where biometric data is captured at long standoff distances, elevated viewing angles, and under adverse atmospheric conditions (e.g., turbulence and high wind velocity). To this end, we propose FarSight, a unified end-to-end system for person recognition that integrates complementary biometric cues across face, gait, and body shape modalities. FarSight incorporates novel algorithms across four core modules: multi-subject detection and tracking, recognition-aware video restoration, modality-specific biometric feature encoding, and quality-guided multi-modal fusion. These components are designed to work cohesively under degraded image conditions, large pose and scale variations, and cross-domain gaps. Extensive experiments on the BRIAR dataset, one of the most comprehensive benchmarks for long-range, multi-modal biometric recognition, demonstrate the effectiveness of FarSight. Compared to our preliminary system, this system achieves a 34.1% absolute gain in 1:1 verification accuracy (TAR@0.1% FAR), a 17.8% increase in closed-set identification (Rank-20), and a 34.3% reduction in open-set identification errors (FNIR@1% FPIR). Furthermore, FarSight was evaluated in the 2025 NIST RTE Face in Video Evaluation (FIVE), which conducts standardized face recognition testing on the BRIAR dataset. These results establish FarSight as a state-of-the-art solution for operational biometric recognition in challenging real-world conditions.

Figures (12)

• Figure 1: Illustration of the IARPA BRIAR whole-body image capture scenarios. (a) Enrollment Indoor Collection: High-quality still images and videos captured from multiple viewpoints under controlled conditions. (b) Probe Outdoor Collection: Videos captured in outdoor environments at varying distances and elevation angles, with challenging factors such as atmospheric turbulence. These settings reflect the real-world conditions encountered in long-range biometric recognition. Permission granted by the subject for use of imagery in publications.
• Figure 2: Overview of the proposed FarSight system, which comprises four modules: (i) multi-subject detection and tracking, (ii) recognition-aware image restoration, (iii) modality-specific encoding for face, gait, and body shape, and (iv) quality-guided multi-modal biometric fusion.
• Figure 3: Overview of the multi-subject detection and tracking in FarSight. A dual-detector approach combines BPJDetbpjdet for body-face localization and YOLOv8yolov8_ultralytics for false positive suppression. Detected subjects are then associated across frames using PSR-ByteTrack zhang2022bytetrack, which refines ByteTrack outputs through patch similarity-based retrieval and track ID correction. This ensures consistent tracking under occlusions, subject re-entry, and long-range degradation.
• Figure 4: Training pipeline for the proposed restoration-recognition co-optimization framework. A distillation loss between siamese-twin models and our face recognition model helps us define a loss for the face recognition model. As shown, not all frames may have detections and only frames with detections are used in $\mathcal{L}_{\text{adaface}}$.
• Figure 5: Illustration of keypoint relative position encoding (KP-RPE) kim2024keypoint. In standard RPE, the attention offset bias is computed based on the distance between the query $Q$ and the key $K$. In KP-RPE, the RPE mechanism is further enhanced by incorporating facial keypoint locations, allowing the RPE to dynamically adjust to the orientation and alignment of the image.