SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment

TL;DR

SEPS tackles fine-grained cross-modal alignment by addressing patch redundancy and ambiguity arising from information density gaps between dense visual patches and sparse textual descriptions. It introduces a two-stage mechanism that fuses unified semantics from dense MLLM-generated text and original sparse captions, guiding patch selection with the SDTPS module and refining patch-word interactions via the HRPA module. The methodology combines dense-text guidance, differentiable patch selection, and relevance-aware aggregation to produce more precise visual-language correspondences, achieving state-of-the-art results on Flickr30K and MS-COCO across multiple backbones, with notable gains in text-to-image and image-to-text retrieval. This approach highlights the practical value of integrating dense textual supervision at the input level to enhance cross-modal alignment and retrieval performance, while maintaining end-to-end trainability and robustness across architectures.

Abstract

Fine-grained cross-modal alignment aims to establish precise local correspondences between vision and language, forming a cornerstone for visual question answering and related multimodal applications. Current approaches face challenges in addressing patch redundancy and ambiguity, which arise from the inherent information density disparities across modalities. Recently, Multimodal Large Language Models (MLLMs) have emerged as promising solutions to bridge this gap through their robust semantic generation capabilities. However, the dense textual outputs from MLLMs may introduce conflicts with the original sparse captions. Furthermore, accurately quantifying semantic relevance between rich visual patches and concise textual descriptions remains a core challenge. To overcome these limitations, we introduce the Semantic-Enhanced Patch Slimming (SEPS) framework, which systematically addresses patch redundancy and ambiguity. Our approach employs a two-stage mechanism to integrate unified semantics from both dense and sparse texts, enabling the identification of salient visual patches. Additionally, it leverages relevance-aware selection with mean value computation to highlight crucial patch-word correspondences, thereby improving cross-modal similarity assessment. Comprehensive experiments on Flickr30K and MS-COCO datasets validate that SEPS achieves superior performance, surpassing existing approaches by 23\%-86\% in rSum across diverse model architectures, with notable enhancements in text-to-image retrieval scenarios. Our implementation is available at https://github.com/Sweet4tars/seps.git.

Figures (5)

• Figure 1: The motivation of our framework, where blue arrows mean language transformer, and green arrows mean vision transformers. (a) Current works suffer from patch ambiguity and patch redundancy due to the limited semantic guidance. (b) Our framework fuses unified semantic derived from dense and sparse texts to guide visual patch selection, and introduces a relevance-aware selection to improve patch-word alignment, which bridges the semantic gap.
• Figure 2: (a) Overview of our Semantic-Enhanced Patch Slimming(SEPS) Framework for fine-grained cross-modal alignment. Given an image-text pair $(I, T)$, we first use pure Transformer encoders to extract visual patch features and textual word features. Then, we propose the SDTPS module to identify text-relevant patches under the support of dense text generated by MLLMs. Finally, we propose the HRPA module to compute the patch-word alignment score $S(I, T)$. (b)(c) The detailed architecture of the proposed SDTPS and HRPA modules, respectively.
• Figure 3: The retrieval performance of different selection ratios $\rho$, constant coefficients $\lambda_1$ and $\lambda_2$ with various visual encoders on Flickr30K.
• Figure 4: The visualization of visual patch selection with different combinations of sparse text and dense text.
• Figure 5: The visualization of cross-modal alignment results of SEPS.