Clarify: Improving Model Robustness With Natural Language Corrections

TL;DR

Clarify presents an end-to-end system that enables non-experts to correct model misconceptions via natural-language feedback collected after initial supervised training. It leverages CLIP-based image-text similarity to surface and partition data by the described failure and employs a distributionally robust optimization objective to retrain the final classifier while keeping the backbone fixed. Across Waterbirds, CelebA, and ImageNet, Clarify yields significant gains in worst-case/subpopulation accuracy with minimal degradation to overall accuracy, and identifies 31 novel hard subpopulations in ImageNet. The work demonstrates the practicality and scalability of human-in-the-loop, concept-level feedback for robust model deployment and motivates future extensions to larger models and richer feedback modalities.

Abstract

The standard way to teach models is by feeding them lots of data. However, this approach often teaches models incorrect ideas because they pick up on misleading signals in the data. To prevent such misconceptions, we must necessarily provide additional information beyond the training data. Prior methods incorporate additional instance-level supervision, such as labels for misleading features or additional labels for debiased data. However, such strategies require a large amount of labeler effort. We hypothesize that people are good at providing textual feedback at the concept level, a capability that existing teaching frameworks do not leverage. We propose Clarify, a novel interface and method for interactively correcting model misconceptions. Through Clarify, users need only provide a short text description of a model's consistent failure patterns. Then, in an entirely automated way, we use such descriptions to improve the training process. Clarify is the first end-to-end system for user model correction. Our user studies show that non-expert users can successfully describe model misconceptions via Clarify, leading to increased worst-case performance in two datasets. We additionally conduct a case study on a large-scale image dataset, ImageNet, using Clarify to find and rectify 31 novel hard subpopulations.

Figures (11)

• Figure 1: Clarify is an interface for interactively correcting model failures due to spurious correlations. (a) Given a model trained with supervised learning, (b) a human describes consistent failure modes of the model entirely in natural language. (c) We automatically incorporate these descriptions to improve the training process by reweighting the training data based on image-text similarity.
• Figure 2: The Clarify interface enables users to iteratively (A) identify and describe model failures and (B) assess the quality of these descriptions. Users can review image examples of correct and incorrect predictions on one class, such as "square" (A1). Based on observed differences, they can input short, natural language descriptions of model failures, such as "red" squares (A2). The system surfaces feedback by splitting the data using the provided description (B1) and displaying an error score (B2). Users can repeat the process to generate improved descriptions.
• Figure 3: For both datasets, (left) non-experts completed annotation tasks using Clarify in less than 3 minutes on average, and (right) models retrained with non-expert annotations outperformed existing baselines in worst-group accuracy.
• Figure 4: (a) Typical images from the "blond" class of CelebA. Non-experts provided textual feedback corresponding to hard subpopulations of (b) lighter and (c) darker hair colors.
• Figure 5: Representative samples corresponding to nine identified spurious correlations in ImageNet. All images shown are in the ImageNet validation set, and belong to the class shown in the first column. Similarity to the specified text annotation splits separates the "easy" and "hard" examples.