Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scientific and Technological Information Oriented Semantics-adversarial and Media-adversarial Cross-media Retrieval

Ang Li, Junping Du, Feifei Kou, Zhe Xue, Xin Xu, Mingying Xu, Yang Jiang

TL;DR

A scientific and technological information oriented Semantics-adversarial and Media-adversarial Cross-media Retrieval method (SMCR), which minimizes the loss of inter-media semantic consistency in addition to modeling intra-media semantic discrimination, to preserve semantic similarity before and after mapping.

Abstract

Cross-media retrieval of scientific and technological information is one of the important tasks in the cross-media study. Cross-media scientific and technological information retrieval obtain target information from massive multi-source and heterogeneous scientific and technological resources, which helps to design applications that meet users' needs, including scientific and technological information recommendation, personalized scientific and technological information retrieval, etc. The core of cross-media retrieval is to learn a common subspace, so that data from different media can be directly compared with each other after being mapped into this subspace. In subspace learning, existing methods often focus on modeling the discrimination of intra-media data and the invariance of inter-media data after mapping; however, they ignore the semantic consistency of inter-media data before and after mapping and media discrimination of intra-semantics data, which limit the result of cross-media retrieval. In light of this, we propose a scientific and technological information oriented Semantics-adversarial and Media-adversarial Cross-media Retrieval method (SMCR) to find an effective common subspace. Specifically, SMCR minimizes the loss of inter-media semantic consistency in addition to modeling intra-media semantic discrimination, to preserve semantic similarity before and after mapping. Furthermore, SMCR constructs a basic feature mapping network and a refined feature mapping network to jointly minimize the media discriminative loss within semantics, so as to enhance the feature mapping network's ability to confuse the media discriminant network. Experimental results on two datasets demonstrate that the proposed SMCR outperforms state-of-the-art methods in cross-media retrieval.

Scientific and Technological Information Oriented Semantics-adversarial and Media-adversarial Cross-media Retrieval

TL;DR

A scientific and technological information oriented Semantics-adversarial and Media-adversarial Cross-media Retrieval method (SMCR), which minimizes the loss of inter-media semantic consistency in addition to modeling intra-media semantic discrimination, to preserve semantic similarity before and after mapping.

Abstract

Cross-media retrieval of scientific and technological information is one of the important tasks in the cross-media study. Cross-media scientific and technological information retrieval obtain target information from massive multi-source and heterogeneous scientific and technological resources, which helps to design applications that meet users' needs, including scientific and technological information recommendation, personalized scientific and technological information retrieval, etc. The core of cross-media retrieval is to learn a common subspace, so that data from different media can be directly compared with each other after being mapped into this subspace. In subspace learning, existing methods often focus on modeling the discrimination of intra-media data and the invariance of inter-media data after mapping; however, they ignore the semantic consistency of inter-media data before and after mapping and media discrimination of intra-semantics data, which limit the result of cross-media retrieval. In light of this, we propose a scientific and technological information oriented Semantics-adversarial and Media-adversarial Cross-media Retrieval method (SMCR) to find an effective common subspace. Specifically, SMCR minimizes the loss of inter-media semantic consistency in addition to modeling intra-media semantic discrimination, to preserve semantic similarity before and after mapping. Furthermore, SMCR constructs a basic feature mapping network and a refined feature mapping network to jointly minimize the media discriminative loss within semantics, so as to enhance the feature mapping network's ability to confuse the media discriminant network. Experimental results on two datasets demonstrate that the proposed SMCR outperforms state-of-the-art methods in cross-media retrieval.
Paper Structure (21 sections, 11 equations, 2 figures, 4 tables, 1 algorithm)

This paper contains 21 sections, 11 equations, 2 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. related work
  3. Problem Formulation
  4. Method
  5. Feature Mapping Network
  6. Label Prediction
  7. Semantics Preservation
  8. Media Constraint
  9. Feature Mapper
  10. Media Discriminator
  11. Adversarial Learning
  12. Experimental Setup
  13. Research Questions
  14. Dataset
  15. Baselines
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Graphical representation of our proposed SMCR. It consists of two processes that play the minimax game: a feature mapper generating intra-media semantics-discriminative, inter-media semantics-consistent, and intra-semantics media-discriminative representations, and a media discriminator distinguishing the original media of representations.
  • Figure 2: The performance of (a) using text to retrieve image, (b) using image to retrieve text, and (c) the average retrieval performance with various parameter and in Wikipedia dataset.