Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cognitive resilience: Unraveling the proficiency of image-captioning models to interpret masked visual content

Zhicheng Du, Zhaotian Xie, Huazhang Ying, Likun Zhang, Peiwu Qin

TL;DR

This work probes the robustness of image captioning models to occluded visual content by systematically masking images along multiple dimensions and evaluating generated captions against originals with semantic similarity. It analyzes four IC models and uses three masking strategies (ratio, block size, color) across a 60-image, multi-dataset collection, leveraging General Text Embeddings for quantitative assessment. Key findings show that IC models can still produce captions from masked images that resemble originals, but performance degrades nonlinearly with mask extent and can be biased or augmented by color-induced cues. The results advance understanding of visual self-supervised learning and robustness in multimodal systems, with implications for designing masking-based probes and improving robustness in real-world applications.

Abstract

This study explores the ability of Image Captioning (IC) models to decode masked visual content sourced from diverse datasets. Our findings reveal the IC model's capability to generate captions from masked images, closely resembling the original content. Notably, even in the presence of masks, the model adeptly crafts descriptive textual information that goes beyond what is observable in the original image-generated captions. While the decoding performance of the IC model experiences a decline with an increase in the masked region's area, the model still performs well when important regions of the image are not masked at high coverage.

Cognitive resilience: Unraveling the proficiency of image-captioning models to interpret masked visual content

TL;DR

This work probes the robustness of image captioning models to occluded visual content by systematically masking images along multiple dimensions and evaluating generated captions against originals with semantic similarity. It analyzes four IC models and uses three masking strategies (ratio, block size, color) across a 60-image, multi-dataset collection, leveraging General Text Embeddings for quantitative assessment. Key findings show that IC models can still produce captions from masked images that resemble originals, but performance degrades nonlinearly with mask extent and can be biased or augmented by color-induced cues. The results advance understanding of visual self-supervised learning and robustness in multimodal systems, with implications for designing masking-based probes and improving robustness in real-world applications.

Abstract

This study explores the ability of Image Captioning (IC) models to decode masked visual content sourced from diverse datasets. Our findings reveal the IC model's capability to generate captions from masked images, closely resembling the original content. Notably, even in the presence of masks, the model adeptly crafts descriptive textual information that goes beyond what is observable in the original image-generated captions. While the decoding performance of the IC model experiences a decline with an increase in the masked region's area, the model still performs well when important regions of the image are not masked at high coverage.
Paper Structure (8 sections, 5 figures)

This paper contains 8 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Method
  3. Experiments
  4. Discussion and Conclusion
  5. The overall pipeline of this study
  6. The t-distributed stochastic neighbor embedding (t-SNE) visualization of the dataset
  7. The effect of the masked block's color
  8. The supplementary effect of the mask processing for the IC model

Figures (5)

  • Figure 1: The quantitative and qualitative outcome results of LLaVA. Three line charts represent the colors of masked block, where the left one is black, middle one is gray, and the right one is white.
  • Figure 2: Overall pipeline of this work, where the random mask processing includes three ways (i.e., masked ratio, masked block size, and color), and the pre-trained GTE model are used to deploy the experiment of semantic textual similarity.
  • Figure 3: The distribution of the dataset used in this work
  • Figure 4: The results of the interpretation of images processed with different mask block colors, where the text in red means the differences with the text from the original image. The first column is the original image, and the second through fourth columns are the images processed with black, gray and white masks respectively (image example from the flickr8k dataset and the used IC model is LLaVA).
  • Figure 5: The results of showing mask processing help the IC model output more informative results, where the text in red means it supplements the text from the original image (image example from the flickr8k dataset and the used IC model is LLaVA).