Augmenting Intra-Modal Understanding in MLLMs for Robust Multimodal Keyphrase Generation

TL;DR

AimKP tackles the fragility of multimodal large language models in multimodal keyphrase generation by strengthening intra-modal understanding without sacrificing cross-modal alignment. It introduces Progressive Modality Masking to enforce deeper modality-specific reasoning and Gradient-Based Filtering to prune uninformative masked samples, achieving stable, curriculum-like training. Empirical results on the CMKP dataset show state-of-the-art MKP performance and notable gains in both text-only and image-only scenarios, reducing gaps with single-modality specialists. The approach demonstrates robust intra-modal enhancement and data efficiency, with broader implications for adapting MLLMs to real-world, noisy multimodal tasks.

Abstract

Multimodal keyphrase generation (MKP) aims to extract a concise set of keyphrases that capture the essential meaning of paired image-text inputs, enabling structured understanding, indexing, and retrieval of multimedia data across the web and social platforms. Success in this task demands effectively bridging the semantic gap between heterogeneous modalities. While multimodal large language models (MLLMs) achieve superior cross-modal understanding by leveraging massive pretraining on image-text corpora, we observe that they often struggle with modality bias and fine-grained intra-modal feature extraction. This oversight leads to a lack of robustness in real-world scenarios where multimedia data is noisy, along with incomplete or misaligned modalities. To address this problem, we propose AimKP, a novel framework that explicitly reinforces intra-modal semantic learning in MLLMs while preserving cross-modal alignment. AimKP incorporates two core innovations: (i) Progressive Modality Masking, which forces fine-grained feature extraction from corrupted inputs by progressively masking modality information during training; (ii) Gradient-based Filtering, that identifies and discards noisy samples, preventing them from corrupting the model's core cross-modal learning. Extensive experiments validate AimKP's effectiveness in multimodal keyphrase generation and its robustness across different scenarios.

Figures (10)

• Figure 1: Examples of MKP, demonstrating cases of image-text aligned (left) and image-text misaligned (right) pairs.
• Figure 2: Performance comparison of MLLMs fine-tuned on multimodal vs. unimodal contexts across three input settings: full multimodal input, text-only input, and image-only input, with metrics: F1@1, F1@3, MAP@5.
• Figure 3: Kernel density plot of cosine similarity of gradients vs. perplexity increase (the ratio of masked-sample keyphrase perplexity to original-sample perplexity).
• Figure 4: The Framework of AimKP. (a) Standard multimodal fine-tuning. (b) Our intra-modal enhancement framework, which (c) progressively masks modality information at increasing rates to force the model to reason deeply within one modality, and (d) dynamically prunes masked samples based on their gradients, preventing conflicting signals from harmful corruptions.
• Figure 5: Case study comparing keyphrase outputs of MM-MKP, LLaVA, and LLaVA-AimKP on four examples.