Multi-aspect Knowledge Distillation with Large Language Model

TL;DR

This work tackles the challenge of fine-grained image classification by moving beyond class-label supervision to transfer diverse, abstract knowledge through multi-aspect distillation. It generates multi-aspect questions with an LLM, extracts corresponding logits from an MLLM, and expands the student’s output to distill these aspect logits using a MaKD loss in addition to standard cross-entropy. Empirical results across six fine-grained and two coarse-grained datasets show consistent improvements over baselines, with ablations confirming the benefits of binary cross-entropy for MaKD and the value of more aspects. The approach also demonstrates extensibility to traditional knowledge distillation, object detection, and limited-data settings, highlighting a practical path to inject visual and contextual understanding from language models into vision systems.

Abstract

Recent advancements in deep learning have significantly improved performance on computer vision tasks. Previous image classification methods primarily modify model architectures or add features, and they optimize models using cross-entropy loss on class logits. Since they focus on classifying images with considering class labels, these methods may struggle to learn various \emph{aspects} of classes (e.g., natural positions and shape changes). Rethinking the previous approach from a novel view, we propose a multi-aspect knowledge distillation method using Multimodal Large Language Models (MLLMs). Our approach involves: 1) querying Large Language Model with multi-aspect questions relevant to the knowledge we want to transfer to the model, 2) extracting corresponding logits from MLLM, and 3) expanding the model's output dimensions to distill these multi-aspect logits. We then apply cross-entropy loss to class logits and binary cross-entropy loss to multi-aspect logits. Through our method, the model can learn not only the knowledge about visual aspects but also the abstract and complex aspects that require a deeper understanding. We primarily apply our method to image classification, and to explore the potential for extending our model, such as object detection. In all experimental results, our method improves the performance of the baselines. Additionally, we analyze the effect of multi-aspect knowledge distillation. These results demonstrate that our method can transfer knowledge about various aspects to the model and the aspect knowledge can enhance model performance in computer vision tasks.

Figures (6)

• Figure 1: Multi-aspect question generation and logit extraction. For multi-aspect question generation (a), we generate various aspect questions from the LLM by using the class and prompt as instructions. For logit extraction about multi-aspect questions (b), we input the generated multi-aspect questions along with the image into the MLLM to extract logits and obtain the probabilities corresponding to yes token. For multi-aspect logit distillation (c), we expand the model's output dimensions and apply both cross-entropy loss and binary cross-entropy loss.
• Figure 2: Ablation study on the number of multi-aspect questions. The x-axis represents the number of aspects (0 represents the baseline model), while the y-axis indicates the accuracy. We run each experiment three times and report the average results.
• Figure 3: Ablation study on weight to the multi-aspect knowledge distillation loss.$\alpha$ value is the weighting factor of multi-aspect logit loss.
• Figure 4: Visualization of the average logit distribution for classes related to aspects. The x-axis represents the classes, and the y-axis represents the mean of the aspect probability distribution from MLLM in the dataset.
• Figure 5: Visualization of t-SNE embeddings for the datasets by aspects. Ours is t-SNE visualizations of the aspect logits from our model (ResNet18), while MLLM is t-SNE visualizations of the aspect logits from the MLLM (InternVL2-8B). The yellow points indicate that the probability of "yes" is close to 1, and the purple points indicate that the probability of "yes" is close to 0. Our model exhibit a similar trend to the aspect logits of the MLLM in fine-grained datasets parkhi2012catsnilsback2008automated.