S$^{5}$Mars: Semi-Supervised Learning for Mars Semantic Segmentation

TL;DR

This work tackles Mars terrain semantic segmentation by addressing data scarcity with the S$^{5}$Mars dataset of 6K high‑resolution, sparsely annotated images across 9 categories, and by proposing a Mars‑specific semi‑supervised learning framework. The method introduces AugIN and SAM‑Mix augmentations to adapt consistency regularization to Mars data, along with a soft‑to‑hard consistency strategy that leverages both soft and hard pseudo‑labels based on confidence. Extensive experiments demonstrate that the proposed approach outperforms state‑of‑the‑art SSL methods on Mars segmentation across different labeled data regimes, supported by comprehensive ablations. The dataset and framework collectively advance reliable, efficient semantic understanding of the Martian surface, with direct implications for rover planning and autonomous navigation in planetary missions.

Abstract

Deep learning has become a powerful tool for Mars exploration. Mars terrain semantic segmentation is an important Martian vision task, which is the base of rover autonomous planning and safe driving. However, there is a lack of sufficient detailed and high-confidence data annotations, which are exactly required by most deep learning methods to obtain a good model. To address this problem, we propose our solution from the perspective of joint data and method design. We first present a newdataset S5Mars for Semi-SuperviSed learning on Mars Semantic Segmentation, which contains 6K high-resolution images and is sparsely annotated based on confidence, ensuring the high quality of labels. Then to learn from this sparse data, we propose a semi-supervised learning (SSL) framework for Mars image semantic segmentation, to learn representations from limited labeled data. Different from the existing SSL methods which are mostly targeted at the Earth image data, our method takes into account Mars data characteristics. Specifically, we first investigate the impact of current widely used natural image augmentations on Mars images. Based on the analysis, we then proposed two novel and effective augmentations for SSL of Mars segmentation, AugIN and SAM-Mix, which serve as strong augmentations to boost the model performance. Meanwhile, to fully leverage the unlabeled data, we introduce a soft-to-hard consistency learning strategy, learning from different targets based on prediction confidence. Experimental results show that our method can outperform state-of-the-art SSL approaches remarkably. Our proposed dataset is available at https://jhang2020.github.io/S5Mars.github.io/.

Figures (11)

• Figure 1: Examples for each label category (highlighted in red).
• Figure 2: Numerical statistics on our S$^{5}$Mars dataset. The figures show the richness of the categories from two aspects: distribution of the number of different labels in each image and distribution of label area. Note no image contains 9 labels simultaneously in its annotation, so it is omitted in (a).
• Figure 3: Some image-label examples in different datasets: (a) AI4Mars AI4Mars. Due to the few defined categories, the annotation diversity and adequacy are insufficient. Meanwhile, there are some cases of mislabeling (red box). (b) Mars-Seg li2022stepwise, which gives a complete pixel-level labeling. However, the label can be misleading when different categories mix up with each other (red box). (c) Our dataset S$^{5}$Mars, which provides accurate labeling for regions with high confidence.
• Figure 4: The overview of the proposed framework for semi-supervised Mars semantic segmentation. We adopt a two-branch teacher-student architecture. Two novel augmentations are proposed as strong augmentations, AugIN and SAM-Mix. AugIN exchanges the statistics of the two samples, i.e., mean and standard deviation. SAM-Mix utilizes an off-the-shelf SAM to obtain the object binary masks to perform copy-paste operation, reducing the uncertainty of the augmented images. Finally, the model is optimized according to a soft-to-hard consistency learning strategy, utlizing both the soft labels $\textbf{p}_i^t$ and the hard labels $\textbf{y}_i^t$ based on the confidence.
• Figure 5: Comparison of different augmentations on SSL for Mars segmentation. (a): Identity, (b): Gaussian Noise, (c): CutOut, (d): Gaussian Blur, (e): Hue, (f): Contrast, (g): Equalize, (h): Brightness, (i): CutMix, (j): Dropout. (k) and (l) are the proposed AugIN and SAM-Mix.