Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OS-HGAdapter: Open Semantic Hypergraph Adapter for Large Language Models Assisted Entropy-Enhanced Image-Text Alignment

Rongjun Chen, Chengsi Yao, Jinchang Ren, Xianxian Zeng, Peixian Wang, Jun Yuan, Jiawen Li, Huimin Zhao, Xu Lu

TL;DR

This paper tackles cross-modal text-image alignment by addressing the entropy gap between textual and visual modalities. It introduces OS-HGAdapter, which synergistically combines an LLM-driven synonymous sentence reinforcement module with a dual-path hypergraph adapter to inject multilateral semantic relations into text and image representations, yielding a calibrated Synonyms embedding space. Empirical results on Flickr30K and MS-COCO show substantial gains in text-to-image and image-to-text retrieval and reduce cross-modal asymmetry, achieving state-of-the-art semantic alignment. The approach demonstrates that controlled entropy augmentation, coupled with higher-order relational modeling, can significantly improve cross-modal understanding and retrieval efficiency in large-scale, open-vocabulary settings.

Abstract

Text-image alignment constitutes a foundational challenge in multimedia content understanding, where effective modeling of cross-modal semantic correspondences critically enhances retrieval system performance through joint embedding space optimization. Given the inherent difference in information entropy between texts and images, conventional approaches often show an imbalance in the mutual retrieval of these two modalities. To address this particular challenge, we propose to use the open semantic knowledge of Large Language Model (LLM) to fill for the entropy gap and reproduce the alignment ability of humans in these tasks. Our entropy-enhancing alignment is achieved through a two-step process: 1) a new prompt template that does not rely on explicit knowledge in the task domain is designed to use LLM to enhance the polysemy description of the text modality. By analogy, the information entropy of the text modality relative to the visual modality is increased; 2) A hypergraph adapter is used to construct multilateral connections between the text and image modalities, which can correct the positive and negative matching errors for synonymous semantics in the same fixed embedding space, whilst reducing the noise caused by open semantic entropy by mapping the reduced dimensions back to the original dimensions. Comprehensive evaluations on the Flickr30K and MS-COCO benchmarks validate the superiority of our Open Semantic Hypergraph Adapter (OS-HGAdapter), showcasing 16.8\% (text-to-image) and 40.1\% (image-to-text) cross-modal retrieval gains over existing methods while establishing new state-of-the-art performance in semantic alignment tasks.

OS-HGAdapter: Open Semantic Hypergraph Adapter for Large Language Models Assisted Entropy-Enhanced Image-Text Alignment

TL;DR

This paper tackles cross-modal text-image alignment by addressing the entropy gap between textual and visual modalities. It introduces OS-HGAdapter, which synergistically combines an LLM-driven synonymous sentence reinforcement module with a dual-path hypergraph adapter to inject multilateral semantic relations into text and image representations, yielding a calibrated Synonyms embedding space. Empirical results on Flickr30K and MS-COCO show substantial gains in text-to-image and image-to-text retrieval and reduce cross-modal asymmetry, achieving state-of-the-art semantic alignment. The approach demonstrates that controlled entropy augmentation, coupled with higher-order relational modeling, can significantly improve cross-modal understanding and retrieval efficiency in large-scale, open-vocabulary settings.

Abstract

Text-image alignment constitutes a foundational challenge in multimedia content understanding, where effective modeling of cross-modal semantic correspondences critically enhances retrieval system performance through joint embedding space optimization. Given the inherent difference in information entropy between texts and images, conventional approaches often show an imbalance in the mutual retrieval of these two modalities. To address this particular challenge, we propose to use the open semantic knowledge of Large Language Model (LLM) to fill for the entropy gap and reproduce the alignment ability of humans in these tasks. Our entropy-enhancing alignment is achieved through a two-step process: 1) a new prompt template that does not rely on explicit knowledge in the task domain is designed to use LLM to enhance the polysemy description of the text modality. By analogy, the information entropy of the text modality relative to the visual modality is increased; 2) A hypergraph adapter is used to construct multilateral connections between the text and image modalities, which can correct the positive and negative matching errors for synonymous semantics in the same fixed embedding space, whilst reducing the noise caused by open semantic entropy by mapping the reduced dimensions back to the original dimensions. Comprehensive evaluations on the Flickr30K and MS-COCO benchmarks validate the superiority of our Open Semantic Hypergraph Adapter (OS-HGAdapter), showcasing 16.8\% (text-to-image) and 40.1\% (image-to-text) cross-modal retrieval gains over existing methods while establishing new state-of-the-art performance in semantic alignment tasks.
Paper Structure (19 sections, 18 equations, 10 figures, 4 tables)

This paper contains 19 sections, 18 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cross-modal image-text matching
  4. Prompt Learning
  5. The Proposed Methodology
  6. Problem Formulation
  7. LLM synonymous sentence reinforcement
  8. Hypergraph-Adapter
  9. Experiments
  10. Datasets Descriptions
  11. Evaluation Metrics
  12. Implementation Details
  13. Quantitative comparison
  14. Further Analysis
  15. Hyperparameter analysis
  16. ...and 4 more sections

Figures (10)

  • Figure 1: Comparison of different text-image alignment methods: (a) the regular way for matching the text and image, only following the original information of the modality. (b) Human's cross-modal ability is dominated by information and corrected by past cognition. The process of humans obtaining information from low-entropy modal information to the final result in our brain is entropy-increasing. The brain gradually fits and aligns information by increasing the entropy of memoriesquiroga2005invariant. (c) Our image-text matching method imitates the pathway of entropy increase in the human brain through past memories. It simulates the entropy increase of human brain memory by prompting learning to guide the large model to output synonymous data. The text feature data after entropy increase is used to construct multi-link relationships using a hypergraph to calculate the endogenous relationship weights, and then sent to the similarity matrix for calculation. The blue highlights are the difference between (a) and (c)
  • Figure 2: Overall structure of OS-HGAdapter. Which consists of in five main parts: The Llama enforce text encoder and Text-HGAdapter reanchor the text space, while visual feature extraction and Vision-HGAdapter capture the high-dimensional space of visual features to anchor the visual space. Finally, we integrate these into a Synonyms embedding space and perform calibration.
  • Figure 3: Comparing the information entropy values of images, text, and enhanced text on the two datasets.
  • Figure 4: Results comparison from (a) template-based and adaptive prompt sentences, and (b) our prompt sentence
  • Figure 5: Hypergraph Adapter structure (dashed lines).
  • ...and 5 more figures