Where Does My Model Underperform? A Human Evaluation of Slice Discovery Algorithms

TL;DR

This work investigates whether slice discovery tools help humans identify where ML models underperform by running two state-of-the-art tools (Domino and PlaneSpot), having 15 participants generate hypotheses for 60 slices, and validating those hypotheses on new data. The study finds that these tools can improve hypothesis correctness relative to a naive baseline, and also increases ease of description and the number of images matching user descriptions, but coherence does not reliably predict correctness and there is substantial variation across users. The results motivate design opportunities focused on hypothesis-centered evaluation, real-time validation, and interactive workflows to better assist stakeholders in understanding and addressing model blind spots. Overall, centering users in evaluating slice-discovery tools is crucial for producing reliable, actionable insights about model behavior.

Abstract

Machine learning (ML) models that achieve high average accuracy can still underperform on semantically coherent subsets ("slices") of data. This behavior can have significant societal consequences for the safety or bias of the model in deployment, but identifying these underperforming slices can be difficult in practice, especially in domains where practitioners lack access to group annotations to define coherent subsets of their data. Motivated by these challenges, ML researchers have developed new slice discovery algorithms that aim to group together coherent and high-error subsets of data. However, there has been little evaluation focused on whether these tools help humans form correct hypotheses about where (for which groups) their model underperforms. We conduct a controlled user study (N = 15) where we show 40 slices output by two state-of-the-art slice discovery algorithms to users, and ask them to form hypotheses about an object detection model. Our results provide positive evidence that these tools provide some benefit over a naive baseline, and also shed light on challenges faced by users during the hypothesis formation step. We conclude by discussing design opportunities for ML and HCI researchers. Our findings point to the importance of centering users when creating and evaluating new tools for slice discovery.

Figures (14)

• Figure 1: An overview of our user study in three steps. (Left) We run different slice discovery algorithms to compute high-error slices (subsets of an input dataset). (Middle) We conduct a human subject study where we show the slices from Step 1 to a human subject, who forms a hypothesis (i.e., description of a subgroup where the model underperforms) corresponding to each slice. (Right) We validate each user-generated hypothesis from Step 2 by calculating the model's accuracy on a new sample of images that match the user's description.
• Figure 2: UI screenshots of the class overview (Left) and slice overview (Right). Errors (where the model failed to detect the object) have red borders. (Left, Top) The class overview shows the total number of images in the test set that belong to the class and the model's average accuracy on these images. (Left, Bottom) A random sample of $40$ images from the test set. (Right, Top) The slice overview shows model's average accuracy on the top-$20$ images belonging to the slice and the entire test set. (Right, Bottom) The top-$20$ ordered images that belong to the slice.
• Figure 3: Example photos from the $5$ selected COCO object classes lin2014microsoft.
• Figure 4: Experimental Setup. (Top) We collect users' hypotheses for $60$ total slices. For each of the $3$ algorithms (rows), and for each of the $5$ classes (columns), we compute the top-$4$ slices. We show $2$ out of $5$ classes (and $24$ out of $60$ total slices) in the figure due to space constraints. (Bottom) Each study participant was shown $12$ total slices output by $3$ different algorithms, and saw slices corresponding to a different class for each algorithm. For example, Participant #1 was asked to develop hypotheses for the top-$4$ slices output by PlaneSpot for the train class. The blue boxes on the top panel highlight the slices shown to Participant #1.
• Figure 5: Hypothesis Correctness. The percentage of hypotheses per condition that are "correct" using a performance gap threshold $\tau = 20\%$ with standard error bars. Percentages are calculated for the subset of hypotheses that we have a sufficiently large number of examples (i.e., at least $15$ matching images) to approximate the performance gap. We find that a greater proportion of users' hypotheses from the PlaneSpot and Domino conditions are correct relative to the Baseline condition.