TM-PATHVQA:90000+ Textless Multilingual Questions for Medical Visual Question Answering
Tonmoy Rajkhowa, Amartya Roy Chowdhury, Sankalp Nagaonkar, Achyut Mani Tripathi
TL;DR
This work tackles the lack of hands-free, multilingual VQA support for medical images by introducing TM-PathVQA, a textless dataset with spoken questions in English, German, and French tied to PathVQA images. The dataset is constructed by translating PathVQA questions with SeamlessM4T and pairing them with 70 hours of audio, yielding 98,397 QAs across 5,004 images. A Multi-Modal Learning framework fuses audio and image features through a Transformer encoder to evaluate multiple feature combinations for Yes/No and open-ended questions. Results show that speech-based VQA with Hu-BERT audio features and Faster RCNN image features outperforms text-based baselines across languages, highlighting the practicality of spoken, multilingual medical VQA and providing a foundation for future attention-based MML approaches.
Abstract
In healthcare and medical diagnostics, Visual Question Answering (VQA) mayemergeasapivotal tool in scenarios where analysis of intricate medical images becomes critical for accurate diagnoses. Current text-based VQA systems limit their utility in scenarios where hands-free interaction and accessibility are crucial while performing tasks. A speech-based VQA system may provide a better means of interaction where information can be accessed while performing tasks simultaneously. To this end, this work implements a speech-based VQA system by introducing a Textless Multilingual Pathological VQA (TMPathVQA) dataset, an expansion of the PathVQA dataset, containing spoken questions in English, German & French. This dataset comprises 98,397 multilingual spoken questions and answers based on 5,004 pathological images along with 70 hours of audio. Finally, this work benchmarks and compares TMPathVQA systems implemented using various combinations of acoustic and visual features.