Demonstration of MaskSearch: Efficiently Querying Image Masks for Machine Learning Workflows

TL;DR

MaskSearch addresses the need to query image–mask collections by mask properties to streamline machine-learning workflows. It introduces a Cumulative Histogram Index ($CHI$) and a filter-verification execution framework to efficiently evaluate $CP(mask, roi, (lv, uv))$ predicates, supporting Filter, Top-$k$, and $MASK oMASK$ aggregation queries. The authors provide a GUI-driven interface that hides SQL details and demonstrates scenarios for model debugging, adversarial-detection, and human–model attention alignment, achieving substantial speedups over baseline approaches. The work offers a practical tool for rapid, mask-based retrieval and analysis in image-centric ML pipelines, with direct impact on model debugging and data-centric improvements.

Abstract

We demonstrate MaskSearch, a system designed to accelerate queries over databases of image masks generated by machine learning models. MaskSearch formalizes and accelerates a new category of queries for retrieving images and their corresponding masks based on mask properties, which support various applications, from identifying spurious correlations learned by models to exploring discrepancies between model saliency and human attention. This demonstration makes the following contributions:(1) the introduction of MaskSearch's graphical user interface (GUI), which enables interactive exploration of image databases through mask properties, (2) hands-on opportunities for users to explore MaskSearch's capabilities and constraints within machine learning workflows, and (3) an opportunity for conference attendees to understand how MaskSearch accelerates queries over image masks.

Figures (4)

• Figure 1: An example misclassified image and its model saliency map with the object bounding boxes (blue and yellow boxes). The salient pixels (red pixels in the saliency map) are focused on the background regions. This reveals that the model relies on irrelevant pixels to make the prediction.
• Figure 2: An example workflow of using MaskSearch's GUI in Scenario 1. In Step 1, 146 -> 17 means that the image with a ground truth label 146: Meleagris Ocellata was misclassified as class 17: Panthera Onca.
• Figure 3: Saliency masks before and after a malicious attack on an example image from ImageNet deng2009imagenet. The object of interest in the image is the fish held by the man.
• Figure 4: Comparison of human attention maps and model saliency maps on images from CUB-200-2011 WahCUB_200_2011. The human attention map shows that humans look at the head and tail of the Pomarine Jaeger to classify it, which are the discriminate traits. The model saliency map shows that the model is focusing on the wings instead. This explains why the model misclassifies Pomarine Jaeger as Long Tailed Jaeger.