Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning

TL;DR

This work tackles cross-lingual image captioning by introducing EHAT, a transformer-based decoder augmented with three heterogeneous modules (MHCA, HARN, HCA) to model local and global alignments among images, English, and Chinese captions. By anchoring visual features via VinVL and employing two-stage training (cross-entropy followed by CIDEr-optimized reinforcement learning), EHAT achieves competitive results against monolingual baselines on MSCOCO and demonstrates effective bilingual caption generation. The approach includes two HARN variants to explore language interaction and shows that heterogeneous attention improves cross-lingual alignment and caption coherence. Overall, EHAT provides a compact, single-model solution for simultaneous bilingual image captioning with strong cross-modal and cross-lingual reasoning, paving the way for richer multilingual vision-language systems.

Abstract

Cross-lingual image captioning is a challenging task that requires addressing both cross-lingual and cross-modal obstacles in multimedia analysis. The crucial issue in this task is to model the global and the local matching between the image and different languages. Existing cross-modal embedding methods based on the transformer architecture oversee the local matching between the image region and monolingual words, especially when dealing with diverse languages. To overcome these limitations, we propose an Embedded Heterogeneous Attention Transformer (EHAT) to establish cross-domain relationships and local correspondences between images and different languages by using a heterogeneous network. EHAT comprises Masked Heterogeneous Cross-attention (MHCA), Heterogeneous Attention Reasoning Network (HARN), and Heterogeneous Co-attention (HCA). The HARN serves as the core network and it captures cross-domain relationships by leveraging visual bounding box representation features to connect word features from two languages and to learn heterogeneous maps. MHCA and HCA facilitate cross-domain integration in the encoder through specialized heterogeneous attention mechanisms, enabling a single model to generate captions in two languages. We evaluate our approach on the MSCOCO dataset to generate captions in English and Chinese, two languages that exhibit significant differences in their language families. The experimental results demonstrate the superior performance of our method compared to existing advanced monolingual methods. Our proposed EHAT framework effectively addresses the challenges of cross-lingual image captioning, paving the way for improved multilingual image analysis and understanding.

Figures (9)

• Figure 1: Compared with the existing cross-lingual image captioning schemes (a) and (b), our method (c) is designed on the ensemble captioning model to generate multiple languages, which is more compact and efficient.
• Figure 2: The overview of our method. The proposed Embedded Heterogeneous Attention Transformer (EHAT) model is composed of Masked Heterogeneous Cross-attention (MHCA), Heterogeneous Attention Reasoning Network (HARN) and Heterogeneous Co-attention (HCA). MHCA provides dimension space alignment via the mask function and self-attention to handle both image and language. HARN, as the core of EHAT, provides cross-modal semantic alignment via heterogeneous modeling. HCA is focused on cross-lingual interactions and is associated with subsequent caption generation. More specific details and functions of network implementations are discussed in Section \ref{['sec3']}. The core module HARN and its variants are shown in Fig. \ref{['variants']}. Best view in color.
• Figure 3: There are two variants for HARN (b) and (c) compared with prototype one (a). The MLP includes a combination of different operations such as FC and softmax. The change in Variant 1 is adjusted with anchored visual features via attention. The change in Variant 2 is the direct transposition of linguistic connection to increase interaction. The changes are marked in purple.
• Figure 4: A sample of the experimental dataset. The image has five English and five Chinese descriptions. The consistency of the captions is ensured by translation and manual proofreading.
• Figure 5: Word clouds for the dataset. The font size in the word cloud represents the frequency of word occurrences, with larger font indicating higher frequencies.