VEglue: Testing Visual Entailment Systems via Object-Aligned Joint Erasing
Zhiyuan Chang, Mingyang Li, Junjie Wang, Cheng Li, Qing Wang
TL;DR
This work introduces VEglue, an object-aligned joint erasing framework for testing Visual Entailment systems. By decomposing hypotheses into fine-grained object-description units and aligning them with premise object regions, VEglue generates tests through three metamorphic relations (MR1–MR3) that perturb both modalities in a coherent, object-centric manner. Across SNLI-VE and e-SNLI-VE datasets and four VE systems, VEglue achieves significantly higher Issue Finding Rates and more detected issues than text- or image-centric baselines, while preserving test validity. Retraining VE models with VEglue-generated tests yields substantial improvements on newly generated tests without sacrificing performance on original data, and the authors release a public reproduction package to enable broader adoption and extension to other multimodal tasks.
Abstract
Visual entailment (VE) is a multimodal reasoning task consisting of image-sentence pairs whereby a promise is defined by an image, and a hypothesis is described by a sentence. The goal is to predict whether the image semantically entails the sentence. VE systems have been widely adopted in many downstream tasks. Metamorphic testing is the commonest technique for AI algorithms, but it poses a significant challenge for VE testing. They either only consider perturbations on single modality which would result in ineffective tests due to the destruction of the relationship of image-text pair, or just conduct shallow perturbations on the inputs which can hardly detect the decision error made by VE systems. Motivated by the fact that objects in the image are the fundamental element for reasoning, we propose VEglue, an object-aligned joint erasing approach for VE systems testing. It first aligns the object regions in the premise and object descriptions in the hypothesis to identify linked and un-linked objects. Then, based on the alignment information, three Metamorphic Relations are designed to jointly erase the objects of the two modalities. We evaluate VEglue on four widely-used VE systems involving two public datasets. Results show that VEglue could detect 11,609 issues on average, which is 194%-2,846% more than the baselines. In addition, VEglue could reach 52.5% Issue Finding Rate (IFR) on average, and significantly outperform the baselines by 17.1%-38.2%. Furthermore, we leverage the tests generated by VEglue to retrain the VE systems, which largely improves model performance (50.8% increase in accuracy) on newly generated tests without sacrificing the accuracy on the original test set.