Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integration

TL;DR

The paper tackles Vietnamese Visual Question Answering by proposing a hybrid architecture that merges global image context from BLIP-2 with detailed local features from EfficientNet, while keeping backbones frozen to reduce training costs. A BEiT-3 based multimodal fusion module and a Vietnamese text encoder (BARTpho) enable strong cross-modal reasoning on the ViVQA dataset, achieving 71.04% accuracy on the test set, outperforming several baselines. Key contributions include a novel image embedding strategy that fuses local and global features, a robust text embedding tailored to Vietnamese, and comprehensive ablations that highlight the importance of fusion type and CNN choice. The work advances ViVQA by leveraging frozen pre-trained backbones and a modular fusion framework, with practical implications for efficient multi-modal systems in low-resource languages, while also acknowledging dataset quality as a current limitation and outlining directions for dataset improvement and domain transfer.

Abstract

Visual Question Answering (VQA) has recently emerged as a potential research domain, captivating the interest of many in the field of artificial intelligence and computer vision. Despite the prevalence of approaches in English, there is a notable lack of systems specifically developed for certain languages, particularly Vietnamese. This study aims to bridge this gap by conducting comprehensive experiments on the Vietnamese Visual Question Answering (ViVQA) dataset, demonstrating the effectiveness of our proposed model. In response to community interest, we have developed a model that enhances image representation capabilities, thereby improving overall performance in the ViVQA system. Specifically, our model integrates the Bootstrapping Language-Image Pre-training with frozen unimodal models (BLIP-2) and the convolutional neural network EfficientNet to extract and process both local and global features from images. This integration leverages the strengths of transformer-based architectures for capturing comprehensive contextual information and convolutional networks for detailed local features. By freezing the parameters of these pre-trained models, we significantly reduce the computational cost and training time, while maintaining high performance. This approach significantly improves image representation and enhances the performance of existing VQA systems. We then leverage a multi-modal fusion module based on a general-purpose multi-modal foundation model (BEiT-3) to fuse the information between visual and textual features. Our experimental findings demonstrate that our model surpasses competing baselines, achieving promising performance. This is particularly evident in its accuracy of $71.04\%$ on the test set of the ViVQA dataset, marking a significant advancement in our research area. The code is available at https://github.com/nngocson2002/ViVQA.

Figures (11)

• Figure 1: Architecture of Multi-vision Contextual Attention, which consists of four main modules: the features extraction networks to learn the image (the red-colored shapes) and question (the orange-colored shape) understanding; the multi-branch contextual attention fusion (the blue-colored shape) to fuse visual and textual features; and the classifier (the green-colored shape) to predict the answer.
• Figure 2: The architecture of the BARTPhoBEiT model, which comprise three main modules: the image embedding (the red-colored shape), the question embedding (the orange-colored shape); the multimodal fusion (the blue-colored shape); and the classifier (the green-colored shape) to predict the answer.
• Figure 3: Overview of our architecture, which consists of four main components: (1) the Image Embedding module, (2) the Question Embedding module, (3) the Multi-modal Fusion module, and (4) the Classifier module for predicting the answer.
• Figure 4: The model architecture of Q-Former in the BLIP-2 framework. We utilize the pre-trained Q-Former to extract visual features from the output embeddings of the frozen image encoder.
• Figure 5: Boxplots illustrate the spread of mean values within the visual features for each operation.