Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Dataset

TL;DR

This study targets Brazilian Portuguese image captioning by contrasting native versus automatically translated Flickr30K captions and evaluating Transformer-based vision-encoder-decoder (VED) models alongside large vision-language models (VLMs). It introduces a cross-native-translated evaluation framework, leveraging CLIP-Score for cross-modal alignment and attention maps for interpretability. The results show Swin-based encoders, especially when paired with DistilBERTimbau, offer robust performance across both native and translated data, while native-pretrained models like ViTucano excel on traditional captioning metrics; GPT-4o variants achieve top CLIP-Score but rely on longer, more diverse vocabularies. The work highlights translation-induced biases and domain shift effects, emphasizing the value of cross-modal evaluation for low-resource languages and providing resources for future extension to broader datasets and VLMs.

Abstract

Image captioning (IC) refers to the automatic generation of natural language descriptions for images, with applications ranging from social media content generation to assisting individuals with visual impairments. While most research has been focused on English-based models, low-resource languages such as Brazilian Portuguese face significant challenges due to the lack of specialized datasets and models. Several studies create datasets by automatically translating existing ones to mitigate resource scarcity. This work addresses this gap by proposing a cross-native-translated evaluation of Transformer-based vision and language models for Brazilian Portuguese IC. We use a version of Flickr30K comprised of captions manually created by native Brazilian Portuguese speakers and compare it to a version with captions automatically translated from English to Portuguese. The experiments include a cross-context approach, where models trained on one dataset are tested on the other to assess the translation impact. Additionally, we incorporate attention maps for model inference interpretation and use the CLIP-Score metric to evaluate the image-description alignment. Our findings show that Swin-DistilBERTimbau consistently outperforms other models, demonstrating strong generalization across datasets. ViTucano, a Brazilian Portuguese pre-trained VLM, surpasses larger multilingual models (GPT-4o, LLaMa 3.2 Vision) in traditional text-based evaluation metrics, while GPT-4 models achieve the highest CLIP-Score, highlighting improved image-text alignment. Attention analysis reveals systematic biases, including gender misclassification, object enumeration errors, and spatial inconsistencies. The datasets and the models generated and analyzed during the current study are available in: https://github.com/laicsiifes/transformer-caption-ptbr.

Figures (11)

• Figure 1: Illustration of the architecture on the merge of Swin Transformer and DistilBERTimbau. Swin-DistilBERTimbau is one of the nine VED combinations used in this work. MHA stands for Multi-Head Attention.
• Figure 2: Simplified illustration of the experiment pipeline.
• Figure 3: Illustration of the mean performance drop when switching context, using a bar chart. The chart depicts the mean percentage performance drop through the evaluation metrics. For instance, "Translated-to-Native vs. Native-to-Native" presents the difference between the Native evaluation of models trained on Translated and Native datasets. The numbers over the bars are their mean percentages, and the lines in the middle of the bars are the standard deviations of the mean percentages. A bar color represents each model to favor the visualization.
• Figure 4: Case 1: Example from Flickr30K Translated with high performance on CLIP-Score compared to the uni-modal metrics.
• Figure 5: Case 2: Example from Flickr30K Translated with poor performance on CLIP-Score compared to the uni-modal metrics.