Long-Range Vision-Based UAV-assisted Localization for Unmanned Surface Vehicles

TL;DR

This work addresses USV localization in GNSS-denied marine environments by introducing a UAV-assisted vision-based framework. A UAV with a movable camera detects the USV from a fixed shoreline altitude, and triangulation using pixel-origin azimuth/elevation and datalink range yields a USV pose, which is fused in an Extended Kalman Filter to estimate the USV state in the inertial frame. The detection model uses transfer-learned YOLOv5 detectors trained on a custom USV dataset, achieving a high $mAP=99.05\%$ on held-out data and enabling real-time predictions during MBZIRC-2024 trials. The method is validated in lake and open-sea scenarios, demonstrating drift-free, long-range localization (up to about 500 m) and robust tracking under ocean disturbances, illustrating its potential to complement GPS in GNSS-denied maritime operations. Future work includes sensor fusion with additional modalities and closed-loop experiments to extend robustness in more complex environments.

Abstract

The global positioning system (GPS) has become an indispensable navigation method for field operations with unmanned surface vehicles (USVs) in marine environments. However, GPS may not always be available outdoors because it is vulnerable to natural interference and malicious jamming attacks. Thus, an alternative navigation system is required when the use of GPS is restricted or prohibited. To this end, we present a novel method that utilizes an Unmanned Aerial Vehicle (UAV) to assist in localizing USVs in GNSS-restricted marine environments. In our approach, the UAV flies along the shoreline at a consistent altitude, continuously tracking and detecting the USV using a deep learning-based approach on camera images. Subsequently, triangulation techniques are applied to estimate the USV's position relative to the UAV, utilizing geometric information and datalink range from the UAV. We propose adjusting the UAV's camera angle based on the pixel error between the USV and the image center throughout the localization process to enhance accuracy. Additionally, visual measurements are integrated into an Extended Kalman Filter (EKF) for robust state estimation. To validate our proposed method, we utilize a USV equipped with onboard sensors and a UAV equipped with a camera. A heterogeneous robotic interface is established to facilitate communication between the USV and UAV. We demonstrate the efficacy of our approach through a series of experiments conducted during the ``Muhammad Bin Zayed International Robotic Challenge (MBZIRC-2024)'' in real marine environments, incorporating noisy measurements and ocean disturbances. The successful outcomes indicate the potential of our method to complement GPS for USV navigation.

Figures (19)

• Figure 1: Project concept. A heterogeneous robotic solution for USV localization in a GPS-free marine environment. The system consists of an UAV that searches and detects the USV from a fixed altitude, taking vision-based techniques to extract the USV position in camera images. Subsequently, an EKF is used estimate the final USV's positions that needed for the USV navigation and target tracking.
• Figure 2: Proposed block diagram of the system. The UAV searches and detects the USV from a fixed altitude. The camera and datalink input are used for USV position estimation, that are required for USV navigation in the marine environment.
• Figure 3: Steps to obtain a fined-tuned USV detection model in real marine environment.
• Figure 4: Dataset for USV detection collected from the UAV camera. The dataset contains USV images from different angles and distances.
• Figure 5: USV detection using YoloV5 model on the UAV's camera images during the final competition MBZIRC-2024 challenge.