CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension

TL;DR

This work targets referring expression comprehension with commonsense knowledge (KB-Ref) by introducing CK-Transformer, a multimodal transformer that integrates top-$K$ KB facts into object representations to improve target localization. The method comprises an image-based fact search that selects facts via a cross-modal similarity and a CK-T architecture with a bi-modal encoder and a fact-aware classifier to fuse expressions, visuals, and facts. It achieves state-of-the-art results on KB-Ref with top-$K$=3 facts, and analyses show that integrating visual information into fact search enhances fact relevance; beyond KB-Ref, introducing facts also benefits traditional REC datasets, though gains depend on dataset characteristics. The approach demonstrates the practical value of incorporating commonsense knowledge into multimodal reasoning for robust referring expression comprehension and related tasks.

Abstract

The task of multimodal referring expression comprehension (REC), aiming at localizing an image region described by a natural language expression, has recently received increasing attention within the research comminity. In this paper, we specifically focus on referring expression comprehension with commonsense knowledge (KB-Ref), a task which typically requires reasoning beyond spatial, visual or semantic information. We propose a novel framework for Commonsense Knowledge Enhanced Transformers (CK-Transformer) which effectively integrates commonsense knowledge into the representations of objects in an image, facilitating identification of the target objects referred to by the expressions. We conduct extensive experiments on several benchmarks for the task of KB-Ref. Our results show that the proposed CK-Transformer achieves a new state of the art, with an absolute improvement of 3.14% accuracy over the existing state of the art.

Figures (5)

• Figure 1: CK-Transformer. For each candidate (the first one in the figure), given an expression, a set of visual region candidates and top-K facts (K$=$3 in the figure), the model first encodes the expression and all top-K facts into corresponding multi-modal features, then fuses these features and maps them into a matching score for the candidate.
• Figure 2: Fine-grained analysis. all: the total number of samples in the test set; with fact: the number of test samples that CK-T predicts correctly; without fact: the number of test samples that CK-T-nf predicts correctly.
• Figure 3: Accuracy across a varying number of facts (top-K).
• Figure 4: Accuracy across a varying number of fact-aware classifier block (M).
• Figure 5: Example fact search process (using the top-1 fact) for different search methods: CK-T (green), CK-T-Uw/oImage (orange) and CK-T-Word2Vec (yellow).