Enhanced Textual Feature Extraction for Visual Question Answering: A Simple Convolutional Approach

TL;DR

This work questions whether complex textual encoders are always optimal for Visual Question Answering and experimentally compares Transformer-based encoders with simpler text models on the VQA-v2 dataset. It introduces ConvGRU, a lightweight text encoder that integrates convolutional n-gram features with a GRU, preserving local textual cues while keeping computational costs low. Across extensive experiments, ConvGRU demonstrates modest gains, particularly for counting-related questions, and strong evidence that local-feature modeling can outperform global-sequence approaches like Transformers in VQA. The findings highlight the practical value of lightweight, multi-scale text representations in multimodal tasks, suggesting that improvements in image encoding and fusion could further amplify gains.

Abstract

Visual Question Answering (VQA) has emerged as a highly engaging field in recent years, with increasing research focused on enhancing VQA accuracy through advanced models such as Transformers. Despite this growing interest, limited work has examined the comparative effectiveness of textual encoders in VQA, particularly considering model complexity and computational efficiency. In this work, we conduct a comprehensive comparison between complex textual models that leverage long-range dependencies and simpler models focusing on local textual features within a well-established VQA framework. Our findings reveal that employing complex textual encoders is not always the optimal approach for the VQA-v2 dataset. Motivated by this insight, we propose ConvGRU, a model that incorporates convolutional layers to improve text feature representation without substantially increasing model complexity. Tested on the VQA-v2 dataset, ConvGRU demonstrates a modest yet consistent improvement over baselines for question types such as Number and Count, which highlights the potential of lightweight architectures for VQA tasks, especially when computational resources are limited.

Figures (4)

• Figure 1: Overview of the VQA architecture highlighting the ConvGRU module. Questions are embedded and processed through convolutional layers to capture local semantics, then integrated via a GRU unit. Image features are extracted using ResNet-101 with spatial attention focusing on key areas. The fusion module combines image and text information, and the classifier outputs the answer. Counting, attention and classifier modules follow the configurations of.
• Figure 2: Schematic representation of the textual modality feature extraction model. Word embeddings are processed through dropout, Tanh activation, padding, and convolutional layers to capture multi-scale features. A residual unit is applied to ensure even questions with subtle n-gram features retain robust initial representations. Then, features are concatenated and fed into a GRU to extract sequential semantics. The inset details the GRU's internal gating mechanisms.
• Figure 3: Distribution of question lengths within the VQA-v2 dataset. Red bars represent the total number of questions at each length, while blue bars show the number of counting questions that start with 'How many'.
• Figure 4: Qualitative results on validation samples indicating how GRU models enhanced with convolutional layer can predict more accurate answers.