A Practical Framework of Key Performance Indicators for Multi-Robot Lunar and Planetary Field Tests

TL;DR

The paper tackles the challenge of comparing heterogeneous multi-robot field trials for lunar prospecting by deriving a scenario-driven KPI framework anchored in three science-focused missions: ilmenite, REE, and water ice. It formalizes KPI categories—Efficiency, Robustness, and Precision—and details metrics that tie mission performance to scientific objectives, including mappings like $A_{mapped}$ and $d_{tot}$, as well as resources detection measures such as the Ratio of Identified Resources. The framework emphasizes scenario- and phase-specific KPI relevance, highlights practical measurement approaches (especially for efficiency and robustness via state logging), and discusses trade-offs and limitations, notably the need for ground-truth data for precision metrics. Validation in a real multi-robot field deployment demonstrates practicality for efficiency- and robustness-related KPIs, while underscoring challenges in obtaining reliable ground-truth for precision metrics. Overall, the framework offers a practical, scalable standard for cross-mission evaluation and targeted technology development in future planetary exploration.

Abstract

Robotic prospecting for critical resources on the Moon, such as ilmenite, rare earth elements, and water ice, requires robust exploration methods given the diverse terrain and harsh environmental conditions. Although numerous analog field trials address these goals, comparing their results remains challenging because of differences in robot platforms and experimental setups. These missions typically assess performance using selected, scenario-specific engineering metrics that fail to establish a clear link between field performance and science-driven objectives. In this paper, we address this gap by deriving a structured framework of KPI from three realistic multi-robot lunar scenarios reflecting scientific objectives and operational constraints. Our framework emphasizes scenario-dependent priorities in efficiency, robustness, and precision, and is explicitly designed for practical applicability in field deployments. We validated the framework in a multi-robot field test and found it practical and easy to apply for efficiency- and robustness-related KPI, whereas precision-oriented KPI require reliable ground-truth data that is not always feasible to obtain in outdoor analog environments. Overall, we propose this framework as a common evaluation standard enabling consistent, goal-oriented comparison of multi-robot field trials and supporting systematic development of robotic systems for future planetary exploration.

Figures (5)

• Figure 1: Scenario 1 (Ilmenite). (a) LROC (Lunar Reconnaissance Orbiter) low-incidence image mosaic with landing zone (LZ), preliminary prospecting grid (white), and elevation transect (orange, red, blue, purple). (b) Elevation profile along the transect. (c–f) Selected orbital maps showcasing thermophysical, topographic, and compositional properties, with the LZ indicated by a white circle.
• Figure 2: General timeline of the mission concept, indicating relevant KPI focus for each phase.
• Figure 3: Scenario 2 (KREEP). (a) LROC low-incidence image mosaic with landing zone (LZ), preliminary prospecting grid (white), and elevation transect (blue, red, orange). (b) Elevation profile along the transect. (c–f) Selected orbital maps showcasing thermophysical, topographic, and compositional properties, with the LZ indicated by a white circle.
• Figure 4: Scenario 3 (Water ice). (a) LROC low-incidence image mosaic with landing zone (LZ), preliminary prospecting grid (white), and elevation transect (orange, red, blue). (b) Elevation profile along the transect. (c–f) Selected orbital maps showcasing thermophysical and topographic properties, with the LZ indicated by a white circle.
• Figure 5: Heterogeneous robotic team during the lunar-analog field deployment used to assess the practical applicability of the proposed KPI framework.