Categorical Knowledge Fused Recognition: Fusing Hierarchical Knowledge with Image Classification through Aligning and Deep Metric Learning

TL;DR

CKFR addresses the tendency of image classifiers to rely on non-target cues by fusing hierarchical category knowledge with deep classifiers through latent-space alignment. It introduces a quantitative-relativity triplet loss that aligns model latent distances with inter-class knowledge distances in a tree-structured knowledge space, enabling end-to-end training without an explicit margin. The method improves both standard classification and weakly-supervised object localization across CIFAR, Mini-ImageNet, and ImageNet-1K, with notable gains in WSOL metrics and more meaningful Grad-CAM explanations. This approach offers a principled way to incorporate structured semantic knowledge into visual recognition, potentially boosting robustness in real-world, out-of-distribution settings.

Abstract

Image classification is a fundamental computer vision task and an important baseline for deep metric learning. In decades efforts have been made on enhancing image classification accuracy by using deep learning models while less attention has been paid on the reasoning aspect of the recognition, i.e., predictions could be made because of background or other surrounding objects rather than the target object. Hierarchical knowledge about image categories depicts inter-class similarities or dissimilarities. Effective fusion of such knowledge with deep learning image classification models is promising in improving target object identification and enhancing the reasoning aspect of the recognition. In this paper, we propose a novel deep metric learning based method to effectively fuse prior knowledge about image categories with mainstream backbone image classification models and enhance the reasoning aspect of the recognition in an end-to-end manner. Existing deep metric learning incorporated image classification methods mainly focus on whether sampled images are from the same class. A new triplet loss function term that aligns distances in the model latent space with those in knowledge space is presented and incorporated in the proposed method to facilitate the dual-modality fusion. Extensive experiments on the CIFAR-10, CIFAR-100, Mini-ImageNet, and ImageNet-1K datasets evaluated the proposed method, and results indicate that the proposed method is effective in enhancing the reasoning aspect of image recognition in terms of weakly-supervised object localization performance.

Figures (6)

• Figure 1: Illustration of the proposed method that identifies the target object rather than background or surrounding objects in the recognized image for enhancing the reasoning aspect of image recognition and improving prediction robustness.
• Figure 2: The training process of the proposed method is illustrated by using a single triplet as an example. The initial step samples the triplet. The following step calculates pseudo-distances between sample categories in the knowledge tree. In the third step, images are encoded to latent representations by the feature extractor. In the next step, distances between sample latent representations are measured, and the proposed quantitative-relativity triplet loss is applied to align the knowledge space with the model latent space by deep metric learning. A cross-entropy loss is also applied for supervised image classification learning.
• Figure 3: Latent space visualizations of the CvT models produced by (a) the baseline and (b) proposed method compared.
• Figure 4: Classification accuracies produced by applying varied $\alpha$ hyperparameter values. Baseline indicates baseline performance without applying the proposed method.
• Figure 5: Grad-CAM visualizations generated by applying varied $\ell$ hyperparameter of the proposed method.