Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction for Visual Grounding

TL;DR

A Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction Visual Grounding method that generates a mask through decoupled sentence phrases, and a text and image hierarchical matching mechanism is constructed, highlighting the role of association between different hierarchies in cross-modal matching.

Abstract

Visual grounding has attracted wide attention thanks to its broad application in various visual language tasks. Although visual grounding has made significant research progress, existing methods ignore the promotion effect of the association between text and image features at different hierarchies on cross-modal matching. This paper proposes a Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction Visual Grounding method. It first generates a mask through decoupled sentence phrases, and a text and image hierarchical matching mechanism is constructed, highlighting the role of association between different hierarchies in cross-modal matching. In addition, a corresponding target object position progressive correction strategy is defined based on the hierarchical matching mechanism to achieve accurate positioning for the target object described in the text. This method can continuously optimize and adjust the bounding box position of the target object as the certainty of the text description of the target object improves. This design explores the association between features at different hierarchies and highlights the role of features related to the target object and its position in target positioning. The proposed method is validated on different datasets through experiments, and its superiority is verified by the performance comparison with the state-of-the-art methods.

Figures (11)

• Figure 1: Main idea of the proposed method. As shown in this figure, the shared objects described in different levels of text are likely to be the target objects. Therefore, the target area will be clearer if shared features are highlighted.
• Figure 2: Architecture of the proposed method. Given the input image and the text, GFCMA extracts features across different modalities and establishes a global relationship between text and image. Meanwhile, HMG parses phrases from the input text and uses them to create masks. Then, these features and masks are sent into CMHM to produce more discriminative features for the referred object. Finally, PPC utilizes the results of hierarchical matching to progressively correct the target object's position, achieving localization of the target object.
• Figure 3: Details of the cross-modal attention layer and the cross-modal global alignment layer. (a) Cross-modal attention layer. (b) Cross-modal global alignment layer. Lin&Sim denotes the linear mapping and similarity measure.
• Figure 4: Details of the cross-modal hierarchical matching (CMHM). (a) Structure of the CMHM . (b) Structure of CMHM-Layer. (c) Structure of hierarchical mask attention (HM Attn).
• Figure 5: Illustration of progressive position correction (PPC) layer. (a) Hierarchical semantic aggregation (HSA). (b) Hierarchical position correction (HPC).