ControlCap: Controllable Region-level Captioning

TL;DR

A controllable region-level captioning approach, which introduces control words to a multimodal model to address the caption degeneration issue, and leverages a discriminative module to generate control words within the caption space to partition it to multiple sub-spaces.

Abstract

Region-level captioning is challenged by the caption degeneration issue, which refers to that pre-trained multimodal models tend to predict the most frequent captions but miss the less frequent ones. In this study, we propose a controllable region-level captioning (ControlCap) approach, which introduces control words to a multimodal model to address the caption degeneration issue. In specific, ControlCap leverages a discriminative module to generate control words within the caption space to partition it to multiple sub-spaces. The multimodal model is constrained to generate captions within a few sub-spaces containing the control words, which increases the opportunity of hitting less frequent captions, alleviating the caption degeneration issue. Furthermore, interactive control words can be given by either a human or an expert model, which enables captioning beyond the training caption space, enhancing the model's generalization ability. Extensive experiments on Visual Genome and RefCOCOg datasets show that ControlCap respectively improves the CIDEr score by 21.6 and 2.2, outperforming the state-of-the-arts by significant margins. Code is available at https://github.com/callsys/ControlCap.

Figures (9)

• Figure 1: An illustration of ControlCap (upper) and a comparison of ControlCap with conventional method (lower). ControlCap introduces interactive controls or self controls (such as fine-grained labels or scene text) to generate specialized captions. To generate less frequent captions, ControlCap requires interactive controls such as "Lamborghini” or "FAFACHL”. For common captions, ControlCap can generate self controls such as "silver, white, car". In the lower figure, the conventional method is challenged by the captioning degradation issue, $i.e.$, predicting the most frequent captions while missing the less frequent ones. In contrast, ControlCap is constrained to generate captions within a few sub-spaces containing the control words so that the opportunity of hitting less frequent captions can be significant.
• Figure 2: Diagram of ControlCap. It comprises visual embedding extraction, control embedding generation, and controllable caption generation. visual embedding extraction consists of a frozen ViT and a contextual visual embedding module, which are introduced to enforce LMM's capacity for region-aware understanding. Control embedding generation consists of a region tagging module and a control embedding module, which are introduced to encode self controls/interactive controls. In controllable caption generation, a bidirectional bridging module maximizes the information exchange between the visual embedding $F_v$ and control embedding $F_c$. The two embeddings are then inputted into a LLM to generate specialized captions.
• Figure 3: Diagram for visual embedding extraction.
• Figure 4: Diagram of control embedding generation during training.
• Figure 5: Diagram of the bidirectional bridging module, which maximizes the information exchange between the visual embedding and control embedding modules.