VibE: A Visual Analytics Workflow for Semantic Error Analysis of CVML Models at Subgroup Level

TL;DR

VibE addresses the challenge of error analysis in CVML models when labels/annotations are scarce by introducing a semantic, subgroup-focused workflow that combines CLIP-based embeddings with GPT-4-driven summaries and concept search. The four-stage VibE pipeline—Model Performance Overview, Semantic Subgroup Identification, Semantic Hypothesis Generation, and Semantic Hypothesis Validation—supports data-centric exploration and hypothesis testing without model internals. The authors implement VibE as a visual analytics system and validate its utility through three CVML case studies and expert interviews, highlighting improved insight generation and hypothesis validation while acknowledging limitations of foundation models and data privacy concerns. Overall, VibE offers practical design guidelines and a flexible, model-agnostic approach to semantic error analysis that can enhance debugging and fairness assessment in CVML deployments.

Abstract

Effective error analysis is critical for the successful development and deployment of CVML models. One approach to understanding model errors is to summarize the common characteristics of error samples. This can be particularly challenging in tasks that utilize unstructured, complex data such as images, where patterns are not always obvious. Another method is to analyze error distributions across pre-defined categories, which requires analysts to hypothesize about potential error causes in advance. Forming such hypotheses without access to explicit labels or annotations makes it difficult to isolate meaningful subgroups or patterns, however, as analysts must rely on manual inspection, prior expertise, or intuition. This lack of structured guidance can hinder a comprehensive understanding of where models fail. To address these challenges, we introduce VibE, a semantic error analysis workflow designed to identify where and why computer vision and machine learning (CVML) models fail at the subgroup level, even when labels or annotations are unavailable. VibE incorporates several core features to enhance error analysis: semantic subgroup generation, semantic summarization, candidate issue proposals, semantic concept search, and interactive subgroup analysis. By leveraging large foundation models (such as CLIP and GPT-4) alongside visual analytics, VibE enables developers to semantically interpret and analyze CVML model errors. This interactive workflow helps identify errors through subgroup discovery, supports hypothesis generation with auto-generated subgroup summaries and suggested issues, and allows hypothesis validation through semantic concept search and comparative analysis. Through three diverse CVML tasks and in-depth expert interviews, we demonstrate how VibE can assist error understanding and analysis.

Figures (8)

• Figure 1: Hierarchical Task Abstraction (HTA) of model error analysis in order to improve model performance using box-and-line notation. We follow the standard conventions for hierarchical task analysis kurniawan2004interaction where tasks are represented by named boxes with a unique ID, which also indicates the hierarchical level of the task. Task abstraction based on lam2017bridging are highlighted as A1-A6. The boxes with dashed lines are out of the scope of this paper. The orange boxes denote the abstract tasks that are relevant to pain points of conducting error analysis.
• Figure 2: Semantic error analysis workflow with four main stages. Each stage aims to answer different questions, which is supported by different views in the user interface. To answer each question, we combine foundation models with visual analytics techniques.
• Figure 3: The data flow of VibE. Based on the original model input/output, we generate auxiliary data about the semantic meanings of model input images. These auxiliary data facilitates the construction of semantic clusters and samples retrieval of semantic concept. Both the original model/data and auxiliary data are presented in the Vibe UI. In practice, the foundation models used in this workflow can be easily replaced with any model that aligns with users' preferences, data compliance needs, or other requirements.
• Figure 4: Image retrieval using natural language. The retrieval pipeline uses a pre-trained CLIP model to retrieve image samples based on a text query.
• Figure 5: VibE provides modal windows to support users to (A) check detailed model input and output, (B) retrieve samples by semantic concept search, and (C) inspect all the samples in one subgroup ranked by specific metric value. (A2) illustrates an example where the model failed to generate views from certain angles. VibE also provides some setting functions to assist users to (D1) filter the semantic search results and (D2) customize the error color encoding.