Transformer-Driven Triple Fusion Framework for Enhanced Multimodal Author Intent Classification in Low-Resource Bangla

TL;DR

This work tackles Bangla multimodal author intent classification by combining text from transformer encoders with visual features from vision backbones. It introduces BangACMM, an intermediate fusion framework that yields a new state-of-the-art macro-F1 of 84.11% on the Uddessho Bangla dataset, outperforming prior Bangla multimodal methods by over 8 points. The results demonstrate that cross-modal interactions at intermediate feature levels substantially enhance intent recognition in low-resource settings, establishing benchmarks and methodological standards for Bangla and other low-resource languages. The study also provides comprehensive ablations and analyses supporting the efficacy of intermediate fusion and cross-modal learning.

Abstract

The expansion of the Internet and social networks has led to an explosion of user-generated content. Author intent understanding plays a crucial role in interpreting social media content. This paper addresses author intent classification in Bangla social media posts by leveraging both textual and visual data. Recognizing limitations in previous unimodal approaches, we systematically benchmark transformer-based language models (mBERT, DistilBERT, XLM-RoBERTa) and vision architectures (ViT, Swin, SwiftFormer, ResNet, DenseNet, MobileNet), utilizing the Uddessho dataset of 3,048 posts spanning six practical intent categories. We introduce a novel intermediate fusion strategy that significantly outperforms early and late fusion on this task. Experimental results show that intermediate fusion, particularly with mBERT and Swin Transformer, achieves 84.11% macro-F1 score, establishing a new state-of-the-art with an 8.4 percentage-point improvement over prior Bangla multimodal approaches. Our analysis demonstrates that integrating visual context substantially enhances intent classification. Cross-modal feature integration at intermediate levels provides optimal balance between modality-specific representation and cross-modal learning. This research establishes new benchmarks and methodological standards for Bangla and other low-resource languages. We call our proposed framework BangACMM (Bangla Author Content MultiModal).

Figures (5)

• Figure 1: Text preprocessing and transformer-based classification pipeline.
• Figure 2: Image preprocessing and vision model architecture pipeline.
• Figure 3: Complete multimodal framework integrating text and image modalities with three fusion strategies.
• Figure 4: Training and validation curves for the best performing model.
• Figure 5: Confusion matrix showing strong classification performance across all six categories.