Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EKF-Based Depth Camera and Deep Learning Fusion for UAV-Person Distance Estimation and Following in SAR Operations

Luka Šiktar, Branimir Ćaran, Bojan Šekoranja, Marko Švaco

TL;DR

The proposed subsystem, designed for use in drones, estimates and measures the distance between the depth camera and the human body keypoints, to maintain the safe distance between the drone and the human target.

Abstract

Search and rescue (SAR) operations require rapid responses to save lives or property. Unmanned Aerial Vehicles (UAVs) equipped with vision-based systems support these missions through prior terrain investigation or real-time assistance during the mission itself. Vision-based UAV frameworks aid human search tasks by detecting and recognizing specific individuals, then tracking and following them while maintaining a safe distance. A key safety requirement for UAV following is the accurate estimation of the distance between camera and target object under real-world conditions, achieved by fusing multiple image modalities. UAVs with deep learning-based vision systems offer a new approach to the planning and execution of SAR operations. As part of the system for automatic people detection and face recognition using deep learning, in this paper we present the fusion of depth camera measurements and monocular camera-to-body distance estimation for robust tracking and following. Deep learning-based filtering of depth camera data and estimation of camera-to-body distance from a monocular camera are achieved with YOLO-pose, enabling real-time fusion of depth information using the Extended Kalman Filter (EKF) algorithm. The proposed subsystem, designed for use in drones, estimates and measures the distance between the depth camera and the human body keypoints, to maintain the safe distance between the drone and the human target. Our system provides an accurate estimated distance, which has been validated against motion capture ground truth data. The system has been tested in real time indoors, where it reduces the average errors, root mean square error (RMSE) and standard deviations of distance estimation up to 15,3\% in three tested scenarios.

EKF-Based Depth Camera and Deep Learning Fusion for UAV-Person Distance Estimation and Following in SAR Operations

TL;DR

The proposed subsystem, designed for use in drones, estimates and measures the distance between the depth camera and the human body keypoints, to maintain the safe distance between the drone and the human target.

Abstract

Search and rescue (SAR) operations require rapid responses to save lives or property. Unmanned Aerial Vehicles (UAVs) equipped with vision-based systems support these missions through prior terrain investigation or real-time assistance during the mission itself. Vision-based UAV frameworks aid human search tasks by detecting and recognizing specific individuals, then tracking and following them while maintaining a safe distance. A key safety requirement for UAV following is the accurate estimation of the distance between camera and target object under real-world conditions, achieved by fusing multiple image modalities. UAVs with deep learning-based vision systems offer a new approach to the planning and execution of SAR operations. As part of the system for automatic people detection and face recognition using deep learning, in this paper we present the fusion of depth camera measurements and monocular camera-to-body distance estimation for robust tracking and following. Deep learning-based filtering of depth camera data and estimation of camera-to-body distance from a monocular camera are achieved with YOLO-pose, enabling real-time fusion of depth information using the Extended Kalman Filter (EKF) algorithm. The proposed subsystem, designed for use in drones, estimates and measures the distance between the depth camera and the human body keypoints, to maintain the safe distance between the drone and the human target. Our system provides an accurate estimated distance, which has been validated against motion capture ground truth data. The system has been tested in real time indoors, where it reduces the average errors, root mean square error (RMSE) and standard deviations of distance estimation up to 15,3\% in three tested scenarios.
Paper Structure (8 sections, 8 equations, 7 figures, 3 tables, 3 algorithms)

This paper contains 8 sections, 8 equations, 7 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Detection and Recognition
  4. Tracking and Following
  5. Kalman Filter-based sensor fusion
  6. Methodology
  7. Results
  8. Conclusion

Figures (7)

  • Figure 1: Camera-to-body (C-B) distance estimation
  • Figure 2: UAV framework and experimental setup for C-B distance evaluation using OptiTrack. (a) Hexsoon EDU450 with vision system and onboard computer. (b) RealSense and motion tracking markers. (c) and (d) Person motion tracking markers.
  • Figure 3: S-H (shoulder-hip) line. S-H line is used to (a) approximate C-B (camera-to-body) distance from monocular image using eq. \ref{['eq:piecewise']}, and (b) extract depth information i.e. C-B distance from the depth camera. (a) S-H line detected on monocular image. (b) S-H line on aligned Depth image.
  • Figure 4: Discrete moving forward and backward: distance and error plots.
  • Figure 5: Continuous moving forward and backward: distance and error plots.
  • ...and 2 more figures