Enhancing Vietnamese VQA through Curriculum Learning on Raw and Augmented Text Representations

TL;DR

This work tackles Vietnamese VQA under high linguistic variability and limited data by integrating paraphrase-based text augmentation with a dynamic curriculum that blends easy (augmented) and hard (raw) samples. A lightweight dual-stream VQA backbone processes both text and image features, while two training branches enable cross-training with augmented questions. The method demonstrates consistent CIDEr gains on OpenViVQA and mixed improvements on ViVQA, with ablations highlighting the importance of scheduler choice, paraphrase count, and backbone compatibility. The approach offers a practical path toward robust, language-specific VQA in low-resource settings and suggests avenues for extending curriculum-based augmentation to other languages and modalities.

Abstract

Visual Question Answering (VQA) is a multimodal task requiring reasoning across textual and visual inputs, which becomes particularly challenging in low-resource languages like Vietnamese due to linguistic variability and the lack of high-quality datasets. Traditional methods often rely heavily on extensive annotated datasets, computationally expensive pipelines, and large pre-trained models, specifically in the domain of Vietnamese VQA, limiting their applicability in such scenarios. To address these limitations, we propose a training framework that combines a paraphrase-based feature augmentation module with a dynamic curriculum learning strategy. Explicitly, augmented samples are considered "easy" while raw samples are regarded as "hard". The framework then utilizes a mechanism that dynamically adjusts the ratio of easy to hard samples during training, progressively modifying the same dataset to increase its difficulty level. By enabling gradual adaptation to task complexity, this approach helps the Vietnamese VQA model generalize well, thus improving overall performance. Experimental results show consistent improvements on the OpenViVQA dataset and mixed outcomes on the ViVQA dataset, highlighting both the potential and challenges of our approach in advancing VQA for Vietnamese language.

Figures (3)

• Figure 1: Overview of our proposed training pipeline utilizing curriculum learning on a mix of raw and augmented samples for a Vietnamese VQA model. The text channel produces embeddings using a text encoder, where augmented embeddings are created by combining the original question embedding with paraphrased embeddings from a paraphrase pool, governed by a threshold dynamically adjusted during training ($t_\text{thresh}$). The image channel extracts visual features using an image encoder in the standard manner. Finally, the text and image features are connected and processed through some fully-connected layers (MLP) to generate the final answer.
• Figure 2: Visualization of training loss (log scale) of the model under different dataset configurations. Baseline uses no augmented samples, Aug 1.0 and Aug 0.8 use fixed thresholds of 1.0 and 0.8 for augmentation, and AugCL 0.8-0.4 applies curriculum learning with thresholds decreasing from 0.8 to 0.4. Training with paraphrased embeddings improves convergence compared to the baseline, with the selected epoch range (11–16) capturing critical optimization stages.
• Figure 3: Comparison of Linear and Cosine Annealing (CA) decay strategies for updating $t_\text{thresh}$ over training epochs. The numbers on each line indicate the number of augmented samples used for training at that epoch. In the experiments in Table \ref{['tab:update_strategy_results']}, Linear decay achieves the best performance when $t_\text{thresh}$ decreases linearly from 1.0 to 0.8. For Cosine Annealing decay, the best performance is achieved when $t_\text{thresh}$ is reduced to 0.6.