Mars-Bench: A Benchmark for Evaluating Foundation Models for Mars Science Tasks

TL;DR

Mars-Bench tackles the lack of standardized benchmarks for Mars science by introducing a comprehensive, ready-to-use benchmark that evaluates foundation models on Martian orbital and surface imagery across 20 datasets for classification, segmentation, and object detection. It standardizes data formats, provides baseline models, and implements a rigorous evaluation protocol with multi-seed, bootstrapped, and normalized reporting to enable fair comparisons. The study reveals that Mars-specific pretraining can outperform general-domain baselines and highlights mixed performance for Earth-observation pretrained models and vision-language models, underscoring the value of domain-adapted pretraining and domain-specific datasets. By offering data, code, and baselines, Mars-Bench aims to accelerate the development of Mars-focused foundation models and advance planetary science through robust, reproducible ML research.

Abstract

Foundation models have enabled rapid progress across many specialized domains by leveraging large-scale pre-training on unlabeled data, demonstrating strong generalization to a variety of downstream tasks. While such models have gained significant attention in fields like Earth Observation, their application to Mars science remains limited. A key enabler of progress in other domains has been the availability of standardized benchmarks that support systematic evaluation. In contrast, Mars science lacks such benchmarks and standardized evaluation frameworks, which have limited progress toward developing foundation models for Martian tasks. To address this gap, we introduce Mars-Bench, the first benchmark designed to systematically evaluate models across a broad range of Mars-related tasks using both orbital and surface imagery. Mars-Bench comprises 20 datasets spanning classification, segmentation, and object detection, focused on key geologic features such as craters, cones, boulders, and frost. We provide standardized, ready-to-use datasets and baseline evaluations using models pre-trained on natural images, Earth satellite data, and state-of-the-art vision-language models. Results from all analyses suggest that Mars-specific foundation models may offer advantages over general-domain counterparts, motivating further exploration of domain-adapted pre-training. Mars-Bench aims to establish a standardized foundation for developing and comparing machine learning models for Mars science. Our data, models, and code are available at: https://mars-bench.github.io/.

Figures (44)

• Figure 1: Representative samples from selected Mars-Bench datasets, from all three task categories.
• Figure 2: Classification Benchmark under Feature Extraction setting: Normalized F1-score of all baselines across six datasets (higher the better). Aggregated plot shows the average over all datasets.
• Figure 3: Segmentation Benchmark under Feature Extraction setting: Normalized IoU of all baselines across six datasets (higher the better). Aggregated plot shows the average over all datasets.
• Figure 4: Object Detection Benchmark under Feature Extraction setting: Normalized mAP of all baselines across three datasets (higher is better). Aggregated plot shows the average over all datasets.
• Figure 5: Classification vs Train size: Normalized F1-score of baselines with a growing size (from 1% to 100%) of the training set. Shaded regions indicate confidence intervals over multiple runs.