Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition

TL;DR

Pixel Distillation extends knowledge distillation to the input level, enabling flexible deployment of vision models on resource-constrained devices by transferring knowledge from a high-resolution, heavy teacher to a low-resolution, lightweight student. It introduces Input Spatial Representation Distillation (ISRD) to map spatial information from large inputs into the student’s input module, and the Teacher-Assistant-Student (TAS) framework to decouple model and input compression, with an Aligned Feature for Preservation (AFP) strategy to adapt detec tion heads. Across image classification and object detection benchmarks, PD and its TAS/AFP extensions consistently improve student performance while reducing input size, model parameters, and MACs relative to the teacher, often outperforming conventional KD and super-resolution baselines. The approach demonstrates strong practical impact for cost-aware deployment on edge devices and heterogeneous architectures (CNNs and ViTs).

Abstract

Previous knowledge distillation (KD) methods mostly focus on compressing network architectures, which is not thorough enough in deployment as some costs like transmission bandwidth and imaging equipment are related to the image size. Therefore, we propose Pixel Distillation that extends knowledge distillation into the input level while simultaneously breaking architecture constraints. Such a scheme can achieve flexible cost control for deployment, as it allows the system to adjust both network architecture and image quality according to the overall requirement of resources. Specifically, we first propose an input spatial representation distillation (ISRD) mechanism to transfer spatial knowledge from large images to student's input module, which can facilitate stable knowledge transfer between CNN and ViT. Then, a Teacher-Assistant-Student (TAS) framework is further established to disentangle pixel distillation into the model compression stage and input compression stage, which significantly reduces the overall complexity of pixel distillation and the difficulty of distilling intermediate knowledge. Finally, we adapt pixel distillation to object detection via an aligned feature for preservation (AFP) strategy for TAS, which aligns output dimensions of detectors at each stage by manipulating features and anchors of the assistant. Comprehensive experiments on image classification and object detection demonstrate the effectiveness of our method. Code is available at https://github.com/gyguo/PixelDistillation.

Figures (10)

• Figure 1: (a) Compared to network architecture, input size has an impact on more kinds of costs, including requirements for cameras and transmission bandwidth. (b) Pixel distillation can provide more flexible cost control schemes for deployment by distilling knowledge across different input sizes and heterogeneous networks.
• Figure 2: Observations about pixel distillation. We report the accuracy (%) of baseline ResNet18 and ViT-Ti/16 under different input sizes in figure b, and the accuracy (%) vs. MACs (G) of our pixel distillation method when the input size is $112\times 112$ (figure a) and $56\times 56$ (figure c), respectively. (b) reports the performance of two student networks under seven input resolutions. (a) and (c) report the performance of our pixel distillation method under two input resolution settings, i.e., $K=2$ and $K=4$. The arrows drawn from (b) represent the baseline performance without knowledge distillation.
• Figure 3: Illustrations of the vanilla PD. (a) vanilla PD consists of a prediction distillation with an input spatial representation distillation (ISRD). ISRD aims to distill spatial information from the large images to train the input module of the student model. (b) The generalized spatial feature preprocess (GSFP) can transfer spatial features from CNNs and ViTs into the same form. (c) Details of the decoder of the ISRD, red text is an example when the backbone is ViT-Ti/16 and the input size of the student is $56\times 56$.
• Figure 4: Study about the distillation position in traditional knowledge distillation ($K$=1) and our pixel distillation ($K$=4) in image classification. The teacher is ResNet50 with $224\times 224$ input, and the student is ResNet18 with $\frac{224}{K}\times \frac{224}{K}$ input.
• Figure 5: Illustrations of the Teacher-Assistant-Student (TAS) framework for image classification task. The assistant model splits the pixel distillation into two stages: the model compression stage which reduces the computational cost by a factor of $\tau$ ($0<\tau<1$) by using lightweight network architecture, and the input compression stage further reduces the input size and computational cost by a factor of $\frac{1}{K^2}$ by using small input.