Interactive Mars Image Content-Based Search with Interpretable Machine Learning

TL;DR

A prototype-based architecture is leveraged to enable users to understand and validate the evidence used by a classifier trained on images from the Mars Science Laboratory (MSL) Curiosity rover mission, and investigate the diversity and correctness of evidence used by the content-based classifier.

Abstract

The NASA Planetary Data System (PDS) hosts millions of images of planets, moons, and other bodies collected throughout many missions. The ever-expanding nature of data and user engagement demands an interpretable content classification system to support scientific discovery and individual curiosity. In this paper, we leverage a prototype-based architecture to enable users to understand and validate the evidence used by a classifier trained on images from the Mars Science Laboratory (MSL) Curiosity rover mission. In addition to providing explanations, we investigate the diversity and correctness of evidence used by the content-based classifier. The work presented in this paper will be deployed on the PDS Image Atlas, replacing its non-interpretable counterpart.

Figures (5)

• Figure 1: Qualitative example of the top-4 most visually similar prototypes for class Mastcam cal target from the MSL surface dataset. Column (a) is the test image, (b) shows the same image overlayed with a heatmap showing regions most activated by the prototype learned during training followed by (c) showing a cropped version of the heatmap after threshold with the similarity score. (d) shows the cropped regions of heatmaps from Column (e). Column (e) shows the training images overlayed with regions obtained after prototypes projection on the training set and . The evidence looks coherent across both training and testing prototypes i.e., column (c) and (d) when the weights are positive.
• Figure 2: Figure showing representative examples from eight classes of the MSL surface Data Set. DRT refers to the Dust Removal Tool aboard the Curiosity rover.
• Figure 3: Explanation for two images from class Sun showing the difference between evidence when the image is misclassified as Night Sky (red, left) vs. when it is classified correctly as Sun (green, right) from a VGG19 backbone. The meaning of the columns is identical to Figure \ref{['fig:mastcam_res']} where (a - e) represents output for the test image in (a) and (f - j) represents output for the test image in (f). Note the prototypes are ordered from most similar to least.
• Figure 4: Comparison of the average prototype diversity over 100 prototypes for the most and least diverse classes, plotted against the position of the prototype based on the order of evidence (sorted based on importance), denoted as $k$, used for classifying MSL surface data. Class Night Sky sees significant improvement while class Float Rock has no improvement in diversity from the inclusion of the diversity loss term.
• Figure 5: Average number of In-class prototypes for top-3 (square) and bottom-3 (triangle) most correct classes vs position of prototype in the order of evidence used $k$ for classifying MSL surface test data by the VGG19 version of ProtoPNet.