Unified Multimodal and Multilingual Retrieval via Multi-Task Learning with NLU Integration
Xinyuan Zhang, Lina Zhang, Lisung Chen, Guangyao Liu, Shuai Nie, Jiaming Xu, Runyu Shi, Ying Huang, Guoquan Zhang
TL;DR
The paper tackles inefficiencies in multimodal retrieval by proposing a unified, multilingual framework that shares a single text encoder across image, short-text, and long-text retrieval and augments it with NLU features. Using a three-stage training process, the model aligns image-text and text-text representations, incorporates intent and slot information, and jointly tunes all components to maximize tri-task performance. Evaluations show strong multilingual image-text retrieval improvements across XTD10 and Multi30K, significant gains in text-to-text retrieval on COCO-QLTI, and improved NLU metrics, validating the approach's ability to reduce storage and compute while enhancing semantic understanding. The proposed method demonstrates practical impact for scalable, multilingual retrieval systems in real-world applications.
Abstract
Multimodal retrieval systems typically employ Vision Language Models (VLMs) that encode images and text independently into vectors within a shared embedding space. Despite incorporating text encoders, VLMs consistently underperform specialized text models on text-only retrieval tasks. Moreover, introducing additional text encoders increases storage, inference overhead, and exacerbates retrieval inefficiencies, especially in multilingual settings. To address these limitations, we propose a multi-task learning framework that unifies the feature representation across images, long and short texts, and intent-rich queries. To our knowledge, this is the first work to jointly optimize multilingual image retrieval, text retrieval, and natural language understanding (NLU) tasks within a single framework. Our approach integrates image and text retrieval with a shared text encoder that is enhanced by NLU features for intent understanding and retrieval accuracy.