MeaCap: Memory-Augmented Zero-shot Image Captioning

TL;DR

A novel Memory-Augmented zero-shot image Captioning framework (MeaCap), equipped with a textual memory, that can generate concept-centered captions that keep high consistency with the image with fewer hallucinations and more world-knowledge.

Abstract

Zero-shot image captioning (IC) without well-paired image-text data can be divided into two categories, training-free and text-only-training. Generally, these two types of methods realize zero-shot IC by integrating pretrained vision-language models like CLIP for image-text similarity evaluation and a pre-trained language model (LM) for caption generation. The main difference between them is whether using a textual corpus to train the LM. Though achieving attractive performance w.r.t. some metrics, existing methods often exhibit some common drawbacks. Training-free methods tend to produce hallucinations, while text-only-training often lose generalization capability. To move forward, in this paper, we propose a novel Memory-Augmented zero-shot image Captioning framework (MeaCap). Specifically, equipped with a textual memory, we introduce a retrieve-then-filter module to get key concepts that are highly related to the image. By deploying our proposed memory-augmented visual-related fusion score in a keywords-to-sentence LM, MeaCap can generate concept-centered captions that keep high consistency with the image with fewer hallucinations and more world-knowledge. The framework of MeaCap achieves the state-of-the-art performance on a series of zero-shot IC settings. Our code is available at https://github.com/joeyz0z/MeaCap.

Figures (9)

• Figure 1: The motivation of our proposed MeaCap where the red is incorrect and green is correct. (a) Training-free methods associate the pie with incorrect location information, which actually get high marks in CLIPscore. This might be due to the fact that CLIP is trained on web-scale noisy image-text data. (b) Existing text-only-training (ToT) methods fail to generate spiderman as some training-free methods do, but the ToT version of our method (MeaCap$_{\mathrm{ToT}}$) can also do that.
• Figure 2: Overview of our proposed MeaCap. The overall data flow is clockwise. i) Given an image, we first retrieve Top-$N$ relevant descriptions from the memory, which is transformed to the subject-predicate-object triplets; we merge and filter nodes to get the key concepts Sec. \ref{['retrieval-then-filter']}. ii) With the memory-augmented visual-related fusion score (Sec. \ref{['fusion score']}), starting from key concepts, the keywords-to-sentence LM can complete the image description by iterative refining (Sec. \ref{['lm']}). $E_i, E_t, S_t$ are CLIP visual encoder, CLIP text encoder, and Sentence-BERT text encoder, respectively. $\bigodot$ denotes the cosine similarity. The $p^{lm}, p^{ITs}, p^{TTs}$ are fluent score in Eq. \ref{['eq:lm']}, image-caption cross-modal similarity Eq. \ref{['eq:visual']}, and memory-caption in-modal similarity Eq. \ref{['eq:memory']}, respectively.
• Figure 3: Examples of zero-shot IC compared with other zero-shot baselines. GT denotes the Ground Truth. ConZIC and ZeroCap are training-free, while MAGIC and DeCap are text-only-training. MeaCap displays the extracted concepts in green and generated caption.
• Figure 4: Examples of real-world knowledge. MeaCap$_\mathrm{ToT}$ can alleviate the world-knowledge-forgotten problem of existing text-only-training methods, such as "batman" in the third image.
• Figure 5: Effect of the number of retrieved memory captions. We reported the performance of $\mathrm{MeaCap_{TF}}$ on the MSCOCO dataset with varying the number of retrieved memory captions.