DEVICE: Depth and Visual Concepts Aware Transformer for OCR-based Image Captioning

TL;DR

DEVICE introduces a depth and visual concepts aware transformer for OCR-based image captioning, addressing limitations of 2D plane reasoning and coarse object descriptions. By incorporating monocular depth maps through a depth-enhanced feature updating module (DeFUM), a salient visual object concepts extractor (VOC), and a semantic-guided alignment module (SgAM), the model achieves richer 3D relational reasoning and more accurate scene text generation. The approach yields state-of-the-art results on TextCaps, notably boosting CIDEr-D to 110.0, and is supported by detailed ablations, qualitative analyses, and convergence improvements. This work advances OCR-based captioning by fusing depth cues with semantic visual concepts to produce more complete and precise captions, with potential for broader multimodal reasoning applications.

Abstract

OCR-based image captioning is an important but under-explored task, aiming to generate descriptions containing visual objects and scene text. Recent studies have made encouraging progress, but they are still suffering from a lack of overall understanding of scenes and generating inaccurate captions. One possible reason is that current studies mainly focus on constructing the plane-level geometric relationship of scene text without depth information. This leads to insufficient scene text relational reasoning so that models may describe scene text inaccurately. The other possible reason is that existing methods fail to generate fine-grained descriptions of some visual objects. In addition, they may ignore essential visual objects, leading to the scene text belonging to these ignored objects not being utilized. To address the above issues, we propose a Depth and Visual Concepts Aware Transformer (DEVICE) for OCR-based image captinong. Concretely, to construct three-dimensional geometric relations, we introduce depth information and propose a depth-enhanced feature updating module to ameliorate OCR token features. To generate more precise and comprehensive captions, we introduce semantic features of detected visual concepts as auxiliary information, and propose a semantic-guided alignment module to improve the model's ability to utilize visual concepts. Our DEVICE is capable of comprehending scenes more comprehensively and boosting the accuracy of described visual entities. Sufficient experiments demonstrate the effectiveness of our proposed DEVICE, which outperforms state-of-the-art models on the TextCaps test set.

Figures (8)

• Figure 1: Our DEVICE significantly improves the quality of captions. In Case a, with depth information and a depth-enhanced feature updating module, DEVICE correctly models 3D relationships between scene texts. With a salient visual object concepts extractor and a semantic-guided alignment module, DEVICE generates more accurate and comprehensive captions, cf. Case b. Scene text is represented in Green, and objects are represented in Blue.
• Figure 2: Overview of Depth and Visual Concepts Aware Transformer. In the reading module, we extract visual concepts along with their depth values. A depth-enhanced feature updating module is constructed to enhance OCR appearance features with the help of depth values. DEVICE utilizes depth information and plane geometry relationships to construct 3D positional relationships. Semantic information of salient visual object concepts promotes fine-grained and holistic captioning, and the proposed semantic-guided alignment module improves the model's efficiency to utilize visual concepts. Words in Blue, Green and Orange represent objects, scene text, and visual concepts, respectively.
• Figure 3: Illustrations for the process of getting the depth values and 3D geometric relationship modeling. The pie chart represents the gray value distribution of object $a^{obj}_n$. We take the gray values with the highest proportion as the depth value (0 means nearest) of the object $a^{obj}_n$ and OCR token $a^{ocr}_m$. $d^{obj}_n$ and $d^{ocr}_m$ denote the depth values of $a^{obj}_n$ and $a^{ocr}_m$, respectively. With depth information, relationships between visual entities become more clear.
• Figure 4: Illustration of the Depth-enhanced Feature Updating Module (DeFUM), which consists of a depth-aware self-attention module and $N$ layers of transformer encoder. The DeFUM could effectively enhance the appearance features of OCR tokens with the help of depth information, which is conducive to constructing the three-dimensional geometric relationships and improving the accuracy of captions.
• Figure 5: Illustration of the Semantic-guided Alignment Module (SgAM), which utilizes semantic attention to transfer the information of visual concepts to semantically aligned OCR Tokens' sub-word embedding. This operation is capable of making model emphasize visual concepts that are highly relevant to the scene text. Besides, SgAM helps multimodal transformer roughly model the spatial location of semantically aligned visual concepts.