Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision Foundation Models for Domain Generalisable Cross-View Localisation in Planetary Ground-Aerial Robotic Teams

Lachlan Holden, Feras Dayoub, Alberto Candela, David Harvey, Tat-Jun Chin

TL;DR

This work tackles the challenge of absolute rover localisation in planetary environments without GPS by matching monocular ground-view images to local aerial maps within a 90° field of view. It introduces a domain-robust pipeline that uses rock segmentation via vision foundation models and a dual-encoder cross-view network trained on synthetic data to bridge the synthetic–real domain gap, with particle filtering to fuse measurements over time. A new cross-view dataset of real rover trajectories and a high-volume synthetic dataset support training and evaluation. Results show accurate localisation can be achieved even when real-labelled data are scarce, and segmentation-based domain adaptation enables practical use on field data, highlighting the approach's potential for scalable planetary autonomy.

Abstract

Accurate localisation in planetary robotics enables the advanced autonomy required to support the increased scale and scope of future missions. The successes of the Ingenuity helicopter and multiple planetary orbiters lay the groundwork for future missions that use ground-aerial robotic teams. In this paper, we consider rovers using machine learning to localise themselves in a local aerial map using limited field-of-view monocular ground-view RGB images as input. A key consideration for machine learning methods is that real space data with ground-truth position labels suitable for training is scarce. In this work, we propose a novel method of localising rovers in an aerial map using cross-view-localising dual-encoder deep neural networks. We leverage semantic segmentation with vision foundation models and high volume synthetic data to bridge the domain gap to real images. We also contribute a new cross-view dataset of real-world rover trajectories with corresponding ground-truth localisation data captured in a planetary analogue facility, plus a high volume dataset of analogous synthetic image pairs. Using particle filters for state estimation with the cross-view networks allows accurate position estimation over simple and complex trajectories based on sequences of ground-view images.

Vision Foundation Models for Domain Generalisable Cross-View Localisation in Planetary Ground-Aerial Robotic Teams

TL;DR

This work tackles the challenge of absolute rover localisation in planetary environments without GPS by matching monocular ground-view images to local aerial maps within a 90° field of view. It introduces a domain-robust pipeline that uses rock segmentation via vision foundation models and a dual-encoder cross-view network trained on synthetic data to bridge the synthetic–real domain gap, with particle filtering to fuse measurements over time. A new cross-view dataset of real rover trajectories and a high-volume synthetic dataset support training and evaluation. Results show accurate localisation can be achieved even when real-labelled data are scarce, and segmentation-based domain adaptation enables practical use on field data, highlighting the approach's potential for scalable planetary autonomy.

Abstract

Accurate localisation in planetary robotics enables the advanced autonomy required to support the increased scale and scope of future missions. The successes of the Ingenuity helicopter and multiple planetary orbiters lay the groundwork for future missions that use ground-aerial robotic teams. In this paper, we consider rovers using machine learning to localise themselves in a local aerial map using limited field-of-view monocular ground-view RGB images as input. A key consideration for machine learning methods is that real space data with ground-truth position labels suitable for training is scarce. In this work, we propose a novel method of localising rovers in an aerial map using cross-view-localising dual-encoder deep neural networks. We leverage semantic segmentation with vision foundation models and high volume synthetic data to bridge the domain gap to real images. We also contribute a new cross-view dataset of real-world rover trajectories with corresponding ground-truth localisation data captured in a planetary analogue facility, plus a high volume dataset of analogous synthetic image pairs. Using particle filters for state estimation with the cross-view networks allows accurate position estimation over simple and complex trajectories based on sequences of ground-view images.
Paper Structure (21 sections, 8 equations, 10 figures, 3 tables)

This paper contains 21 sections, 8 equations, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Planetary navigation and localisation
  4. Deep-learning-based cross-view localisation
  5. Semantic segmentation and object detection
  6. Method
  7. Domain-robust input via rock segmentation
  8. Cross-view embedding via dual-encoder network
  9. Particle filtering for state estimation
  10. Dataset
  11. Laboratory-collected data
  12. Synthetic data
  13. Experiments
  14. Domain-robust rock segmentation
  15. Learning dual-encoder cross-view embeddings
  16. ...and 6 more sections

Figures (10)

  • Figure 1: A high-level overview outlining our method of estimating the state of a rover within an aerial image.
  • Figure 2: Full rock segmentation pipeline using LLMDet and SAM 2.
  • Figure 3: Dual-encoder cross-view localising network structure, adapted from zhu.etal.2022_transgeo. We use $L=12$ encoder blocks.
  • Figure 4: Examples of ground view (bottom) and rectified aerial view (top) images in our planetary analogue dataset.
  • Figure 5: Qualitative examples showing the success of the LLMDet + SAM 2 segmentation on a real Mars image (top left), real planetary analogue image (top right), and synthetic image (bottom).
  • ...and 5 more figures