LVLM-Aided Alignment of Task-Specific Vision Models

TL;DR

The paper tackles the challenge of spurious correlations in small, task-specific vision models used in high-stakes domains. It introduces LVLM-Aided Visual Alignment (LVLM-VA), which uses an LVLM as a Critic & Judge to translate model explanations into language and to convert class-level human specifications into instance-wise feedback, guided by a Right-for-the-Right-Reasons loss. Central to LVLM-VA are PPEPS-WGM based segmentation and a low-entropy sampling strategy to identify and suppress spurious features without fine-grained human annotations, enabling alignment on synthetic and real medical datasets and improving worst-group accuracy while preserving overall performance. The approach demonstrates scalable, human-centered alignment that enhances reliability and trustworthiness in deployments, with potential broad applicability across domains where domain knowledge is critical for robust decision-making.

Abstract

In high-stakes domains, small task-specific vision models are crucial due to their low computational requirements and the availability of numerous methods to explain their results. However, these explanations often reveal that the models do not align well with human domain knowledge, relying instead on spurious correlations. This might result in brittle behavior once deployed in the real-world. To address this issue, we introduce a novel and efficient method for aligning small task-specific vision models with human domain knowledge by leveraging the generalization capabilities of a Large Vision Language Model (LVLM). Our LVLM-Aided Visual Alignment (LVLM-VA) method provides a bidirectional interface that translates model behavior into natural language and maps human class-level specifications to image-level critiques, enabling effective interaction between domain experts and the model. Our method demonstrates substantial improvement in aligning model behavior with human specifications, as validated on both synthetic and real-world datasets. We show that it effectively reduces the model's dependence on spurious features and on group-specific biases, without requiring fine-grained feedback.

Figures (17)

• Figure 1: LVLM-Aided Visual Alignment (LVLM-VA) of a small task-specific vision model steered by human domain knowledge, using Explainable AI (XAI) in conjunction with a Large Vision Language Model (LVLM) Critic & Judge pair. The domain knowledge is induced into the system via human specifications on a class level supporting the LVLM to identify relevant core features within the input images and detect spurious shortcuts based on the model explanations. The Critic & Judge assessment is used to correct the original model in an alignment step but can also be used to provide feedback to the human expert.
• Figure 2: Correction mask generation process by the Critic & Judge pair for a vision model trained on a knee radiograph dataset. The image shows a hospital tag in the bottom left, which the model learned as a shortcut to classify the condition of the knee. All images are of size $224\times 224$.
• Figure 3: Intermediate results for aligning an MLP model for classifying DecoyMNIST. Based on the input of the original image, the segmentation map and the class level description, the LVLM-Critic correctly identifies that the top left corner includes the spurious decoy. The LLM-Judge assigns the right binary label which is subsequently transferred into the correction mask where black refers to 'not relevant'.
• Figure 4: The explanations generated for a test example for an MLP model trained on DecoyMNIST before and after the alignment step. The original model clearly focuses on the spurious decoy in the upper left corner whereas the attribution of the aligned model is almost fully distributed across the actual digit.
• Figure 5: Test set embeddings of the MLP model before (left) and after (right) the LVLM-VA alignment step. The clusters are more separated, as the model is less affected by spurious shortcuts.