Let Video Teaches You More: Video-to-Image Knowledge Distillation using DEtection TRansformer for Medical Video Lesion Detection

TL;DR

The paper addresses the trade-off between image-based and video-based lesion detectors in medical videos by introducing V2I-DETR, a teacher–student DETR framework that distills temporal context from multiple frames into a single-image student. It introduces a Multi-scale Spatiotemporal Interaction module in the teacher, and two distillation mechanisms—Target-guided Feature Distillation and Cross-view Query Distillation—to transfer spatiotemporal and proposal information to the student without increasing inference cost. Across SUN Colonoscopy and Breast Ultrasound datasets, V2I-DETR achieves state-of-the-art or competitive results while maintaining real-time inference at 30 FPS, outperforming existing methods by substantial margins. Ablation studies confirm that MSI, TFD, and CQD each contribute meaningful gains, with Gaussian soft foreground masking and random cross-view sampling providing additional benefits. The approach holds practical potential for robust, fast medical video lesion detection in clinical workflows.

Abstract

AI-assisted lesion detection models play a crucial role in the early screening of cancer. However, previous image-based models ignore the inter-frame contextual information present in videos. On the other hand, video-based models capture the inter-frame context but are computationally expensive. To mitigate this contradiction, we delve into Video-to-Image knowledge distillation leveraging DEtection TRansformer (V2I-DETR) for the task of medical video lesion detection. V2I-DETR adopts a teacher-student network paradigm. The teacher network aims at extracting temporal contexts from multiple frames and transferring them to the student network, and the student network is an image-based model dedicated to fast prediction in inference. By distilling multi-frame contexts into a single frame, the proposed V2I-DETR combines the advantages of utilizing temporal contexts from video-based models and the inference speed of image-based models. Through extensive experiments, V2I-DETR outperforms previous state-of-the-art methods by a large margin while achieving the real-time inference speed (30 FPS) as the image-based model.

Figures (5)

• Figure 1: Motivation of video-to-image knowledge transfer. In the training phase, a novel knowledge distillation with multi-frame collaboration is designed to transfer the rich temporal information to the student. We note the student network only takes a single image as input during inference.
• Figure 2: Pipeline of V2I-DETR. In the training phase, a Multi-scale Spatiotemporal Interaction module (MSI) is designed to enable the interaction of temporal contexts from reference frames. To guide the student model, we adaptively transfer the foreground features from the teacher to the student by Target-Guided Feature Distillation (TFD) and transfer the temporal relations of target proposals to the student Decoder by Cross-View Query Distillation (CQD). During inference, only the student model is employed to conduct image-level prediction. Figure best viewed in color.
• Figure 3: Multi-scale Spatiotemporal Interaction (MSI) Module. Backbone features from different frames interact at each scale, where low-level features are first enhanced by local attention while high-level features are further aggregated by global attention. $k$ is the kernel size.
• Figure 4: Qualitative comparisons. Visualization results produced by our method and state-of-the-art methods on SUN video dataset (left part) and BUV dataset (right part). The green box indicates the correct predictions and the red box denotes the wrong predictions.
• Figure 5: Ablation study. (a) The effect of loss function hyperparameters on detection f1-score. (b) The effect of Training Frame numbers.