Domino: Discovering Systematic Errors with Cross-Modal Embeddings

TL;DR

The paper tackles the problem of identifying underperforming, coherent data slices in high-dimensional domains. It introduces Domino, a cross-modal embedding–based SDM paired with an error-aware Gaussian mixture model to discover and describe coherent slices, including automatic natural language explanations. A scalable evaluation framework across 1,235 slice-discovery settings demonstrates that Domino improves ground-truth slice detection by 12 percentage points over prior methods and can produce NL descriptions for many slices. The approach leverages large-scale cross-modal representations (e.g., CLIP, ConVIRT) to enhance coherence and facilitate human-understandable slice descriptions, with implications for robust model auditing in real-world, black-box settings.

Abstract

Machine learning models that achieve high overall accuracy often make systematic errors on important subsets (or slices) of data. Identifying underperforming slices is particularly challenging when working with high-dimensional inputs (e.g. images, audio), where important slices are often unlabeled. In order to address this issue, recent studies have proposed automated slice discovery methods (SDMs), which leverage learned model representations to mine input data for slices on which a model performs poorly. To be useful to a practitioner, these methods must identify slices that are both underperforming and coherent (i.e. united by a human-understandable concept). However, no quantitative evaluation framework currently exists for rigorously assessing SDMs with respect to these criteria. Additionally, prior qualitative evaluations have shown that SDMs often identify slices that are incoherent. In this work, we address these challenges by first designing a principled evaluation framework that enables a quantitative comparison of SDMs across 1,235 slice discovery settings in three input domains (natural images, medical images, and time-series data). Then, motivated by the recent development of powerful cross-modal representation learning approaches, we present Domino, an SDM that leverages cross-modal embeddings and a novel error-aware mixture model to discover and describe coherent slices. We find that Domino accurately identifies 36% of the 1,235 slices in our framework - a 12 percentage point improvement over prior methods. Further, Domino is the first SDM that can provide natural language descriptions of identified slices, correctly generating the exact name of the slice in 35% of settings.

Figures (9)

• Figure 1: Proposed Approach. (Left) We design an evaluation framework to systematically compare SDMs across diverse slice settings. Here, the example slice setting includes a dataset that displays a strong correlation between the presence of birds and skies. (Right) A classifier trained to detect the presence of birds makes false positive predictions on skies without birds. We present Domino, a novel SDM that uses cross-modal embeddings to identify and describe the error slice.
• Figure 2: Evaluation Framework. We propose a framework for generating slice discovery settings from any base dataset with class structure or metadata.
• Figure 3: Cross-modal embeddings enable accurate slice discovery. Using our evaluation framework, we demonstrate that the use of cross-modal embeddings leads to consistent improvements in slice discovery across three datasets and two input modalities ($1{,}235$ settings).
• Figure 4: Error-aware mixture model enables accurate slice discovery. When cross-modal embeddings are provided as input, our error-aware mixture model often outperforms previously-designed SDMs. Results on medical images and medical time-series data are in Section \ref{['app:add_experiments']}.
• Figure 5: Domino produces natural language descriptions of discovered slices. Natural language descriptions for discovered slices in (top row) 3 settings randomly selected from the set of the 85 rare slice, natural image settings where Domino includes the exact name of the slice in its top 5 slice descriptions; (middle row) 3 settings randomly selected from the set of the 45 correlation slice, natural image settings where Domino includes the exact name of the slice in its top 5 slice descriptions and precision-at-25 exceeds $0.8$; and (bottom row) 3 settings randomly selected from the set of the 95 noisy label slice, natural image settings where Domino includes the exact name of the slice in its top 5 slice descriptions and precision-at-25 exceeds $0.8$. The length of the bars beneath each description are proportional to the dot product score for the description (see Section \ref{['method_sec:explanations']}). Also shown are the top 3-4 images that Domino associates with the discovered slice.