SliceLens: Fine-Grained and Grounded Error Slice Discovery for Multi-Instance Vision Tasks

TL;DR

SliceLens tackles the lack of fine grained, grounded error slice analysis for multi instance vision tasks by presenting a hypothesis driven framework that uses LLMs for hypothesis generation and VLMs for grounded verification. It formulates and tests diverse failure hypotheses as natural language queries, grounding them to regions and validating systematically via a slope based trend analysis linking slice confidence to error rates. The FeSD benchmark provides expert annotated, grounded ground truth slices for detection and segmentation, capturing fine grained and compositional failure modes. Across bias discovery benchmarks and FeSD, SliceLens achieves state of the art precision and demonstrates practical impact by enabling targeted model repairs for both classification and detection tasks.

Abstract

Systematic failures of computer vision models on subsets with coherent visual patterns, known as error slices, pose a critical challenge for robust model evaluation. Existing slice discovery methods are primarily developed for image classification, limiting their applicability to multi-instance tasks such as detection, segmentation, and pose estimation. In real-world scenarios, error slices often arise from corner cases involving complex visual relationships, where existing instance-level approaches lacking fine-grained reasoning struggle to yield meaningful insights. Moreover, current benchmarks are typically tailored to specific algorithms or biased toward image classification, with artificial ground truth that fails to reflect real model failures. To address these limitations, we propose SliceLens, a hypothesis-driven framework that leverages LLMs and VLMs to generate and verify diverse failure hypotheses through grounded visual reasoning, enabling reliable identification of fine-grained and interpretable error slices. We further introduce FeSD (Fine-grained Slice Discovery), the first benchmark specifically designed for evaluating fine-grained error slice discovery across instance-level vision tasks, featuring expert-annotated and carefully refined ground-truth slices with precise grounding to local error regions. Extensive experiments on both existing benchmarks and FeSD demonstrate that SliceLens achieves state-of-the-art performance, improving Precision@10 by 0.42 (0.73 vs. 0.31) on FeSD, and identifies interpretable slices that facilitate actionable model improvements, as validated through model repair experiments.

Figures (14)

• Figure 1: Error slice discovery illustrated on multi-instance tasks (e.g., detection).
• Figure 2: Comparison of error slice benchmark datasets.
• Figure 3: Overview of SliceLens. Illustrated with the “bicycle detection” example.
• Figure 4: Performance comparison across different error categories.
• Figure 5: Comparison between tags and our hypotheses on semantic relevance with ground truth error slices.