Direction-Oriented Visual-semantic Embedding Model for Remote Sensing Image-text Retrieval

TL;DR

A novel direction-oriented visual-semantic embedding model (DOVE) to mine the relationship between vision and language and exploit a global visual-semantic constraint to reduce single visual dependency and serve as an external constraint for the final visual and textual representations.

Abstract

Image-text retrieval has developed rapidly in recent years. However, it is still a challenge in remote sensing due to visual-semantic imbalance, which leads to incorrect matching of non-semantic visual and textual features. To solve this problem, we propose a novel Direction-Oriented Visual-semantic Embedding Model (DOVE) to mine the relationship between vision and language. Our highlight is to conduct visual and textual representations in latent space, directing them as close as possible to a redundancy-free regional visual representation. Concretely, a Regional-Oriented Attention Module (ROAM) adaptively adjusts the distance between the final visual and textual embeddings in the latent semantic space, oriented by regional visual features. Meanwhile, a lightweight Digging Text Genome Assistant (DTGA) is designed to expand the range of tractable textual representation and enhance global word-level semantic connections using less attention operations. Ultimately, we exploit a global visual-semantic constraint to reduce single visual dependency and serve as an external constraint for the final visual and textual representations. The effectiveness and superiority of our method are verified by extensive experiments including parameter evaluation, quantitative comparison, ablation studies and visual analysis, on two benchmark datasets, RSICD and RSITMD.

Figures (12)

• Figure 1: (a): Visual-semantic balance and visual-semantic imbalance. (b): Two main factors cause visual-semantic imbalance, visual-semantic redundancy and inter-class similarity.
• Figure 2: Schematic illustration of DOVE model. Internal constraint with the regional visual embedding as orientation and external boundary of global visual-semantic constraint allows matching visual and textual embeddings to approximate each other in the latent embedding space.
• Figure 3: The use of text backward hidden layer features to mine text forward hidden layer features in DTGA module.
• Figure 4: ROAM module: (a) IFA module; (b) IGA module.
• Figure 5: Insignificant and significant images divided by whether it contains significant objects or not.