Let Multimodal Embedders Learn When to Augment Query via Adaptive Query Augmentation
Wongyu Kim, Hochang Lee, Sanghak Lee, Yoonsung Kim, Jaehyun Park
TL;DR
The paper tackles the inefficiency and potential performance drops from always augmenting queries in multimodal embedding. It introduces M-Solomon, a universal multimodal embedder that adaptively decides when to augment by partitioning the training data into $D_A$ and $D_E$, synthesizing augmentations with a Multimodal LLM, and emitting the tokens /augment or /embed to control inference. The model learns a joint objective $\mathcal{L} = \alpha_{\text{rep}} \mathcal{L}_{\text{rep}} + \alpha_{\text{gen}} \mathcal{L}_{\text{gen}}$ with $\mathcal{L}_{\text{gen}} = - \sum_{t=1}^{T} \log P(g_t \mid q, g_{<t})$ and $\mathcal{L}_{\text{rep}}$ as a contrastive loss over augmented queries and documents, enabling adaptive generation of augmentations only when beneficial. Experiments on the MMEB benchmark show that M-Solomon outperforms both NoAug and AlwaysAug baselines, achieving strong retrieval accuracy with substantially reduced embedding latency and improved OOD generalization. The work demonstrates that selective, synthesis-informed augmentation can enhance multimodal retrieval efficiency and effectiveness in practice.
Abstract
Query augmentation makes queries more meaningful by appending further information to the queries to find relevant documents. Current studies have proposed Large Language Model (LLM)-based embedders, which learn representation for embedding and generation for query augmentation in a multi-task manner by leveraging the generative capabilities of LLM. During inference, these jointly trained embedders have conducted query augmentation followed by embedding, showing effective results. However, augmenting every query leads to substantial embedding latency and query augmentation can be detrimental to performance for some queries. Also, previous methods have not been explored in multimodal environments. To tackle these problems, we propose M-Solomon, a universal multimodal embedder that can adaptively determine when to augment queries. Our approach first divides the queries of the training datasets into two groups at the dataset level. One includes queries that require augmentation and the other includes queries that do not. Then, we introduces a synthesis process that generates appropriate augmentations for queries that require them by leveraging a powerful Multimodal LLM (MLLM). Next, we present adaptive query augmentation. Through this step, M-Solomon can conduct query augmentation only when necessary by learning to generate synthetic augmentations with the prefix /augment for queries that demand them and to generate the simple string /embed for others. Experimental results showed that M-Solomon not only surpassed the baseline without augmentation by a large margin but also outperformed the baseline that always used augmentation, providing much faster embedding latency.