Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Neural ODEs via Knowledge Distillation

Haoyu Chu, Shikui Wei, Qiming Lu, Yao Zhao

TL;DR

This paper addresses the poor performance of Neural ODEs on image recognition by introducing a teacher–student training regimen where ResNet models provide soft targets via knowledge distillation. By combining soft targets with hard labels, Neural ODEs achieve substantially higher accuracy on CIFAR10 and SVHN and exhibit improved robustness to adversarial attacks. The approach also reveals that increasing the time horizon of the ODE integration enhances robustness, supporting the claim that gains arise from richer supervision and not obfuscated gradients. Overall, the method narrows the performance gap between Neural ODEs and discrete networks on complex visual tasks and offers a practical boost in adversarial resilience, with clear trade-offs in training time. The work provides a concrete framework for leveraging discrete teacher models to enrich continuous-depth architectures in image classification.

Abstract

Neural Ordinary Differential Equations (Neural ODEs) construct the continuous dynamics of hidden units using ordinary differential equations specified by a neural network, demonstrating promising results on many tasks. However, Neural ODEs still do not perform well on image recognition tasks. The possible reason is that the one-hot encoding vector commonly used in Neural ODEs can not provide enough supervised information. We propose a new training based on knowledge distillation to construct more powerful and robust Neural ODEs fitting image recognition tasks. Specially, we model the training of Neural ODEs into a teacher-student learning process, in which we propose ResNets as the teacher model to provide richer supervised information. The experimental results show that the new training manner can improve the classification accuracy of Neural ODEs by 24% on CIFAR10 and 5% on SVHN. In addition, we also quantitatively discuss the effect of both knowledge distillation and time horizon in Neural ODEs on robustness against adversarial examples. The experimental analysis concludes that introducing the knowledge distillation and increasing the time horizon can improve the robustness of Neural ODEs against adversarial examples.

Improving Neural ODEs via Knowledge Distillation

TL;DR

This paper addresses the poor performance of Neural ODEs on image recognition by introducing a teacher–student training regimen where ResNet models provide soft targets via knowledge distillation. By combining soft targets with hard labels, Neural ODEs achieve substantially higher accuracy on CIFAR10 and SVHN and exhibit improved robustness to adversarial attacks. The approach also reveals that increasing the time horizon of the ODE integration enhances robustness, supporting the claim that gains arise from richer supervision and not obfuscated gradients. Overall, the method narrows the performance gap between Neural ODEs and discrete networks on complex visual tasks and offers a practical boost in adversarial resilience, with clear trade-offs in training time. The work provides a concrete framework for leveraging discrete teacher models to enrich continuous-depth architectures in image classification.

Abstract

Neural Ordinary Differential Equations (Neural ODEs) construct the continuous dynamics of hidden units using ordinary differential equations specified by a neural network, demonstrating promising results on many tasks. However, Neural ODEs still do not perform well on image recognition tasks. The possible reason is that the one-hot encoding vector commonly used in Neural ODEs can not provide enough supervised information. We propose a new training based on knowledge distillation to construct more powerful and robust Neural ODEs fitting image recognition tasks. Specially, we model the training of Neural ODEs into a teacher-student learning process, in which we propose ResNets as the teacher model to provide richer supervised information. The experimental results show that the new training manner can improve the classification accuracy of Neural ODEs by 24% on CIFAR10 and 5% on SVHN. In addition, we also quantitatively discuss the effect of both knowledge distillation and time horizon in Neural ODEs on robustness against adversarial examples. The experimental analysis concludes that introducing the knowledge distillation and increasing the time horizon can improve the robustness of Neural ODEs against adversarial examples.
Paper Structure (19 sections, 3 equations, 1 figure, 3 tables, 1 algorithm)

This paper contains 19 sections, 3 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. Background
  4. Neural ODEs
  5. Knowledge Distillation
  6. Distilling the Knowledge from ResNet to Neural ODEs
  7. Experimental Setup
  8. The Network Architecture of Neural ODEs
  9. Impementation
  10. Experimental Results
  11. The Training of Neural ODEs without Using Knowledge Distillation
  12. The Training of Neural ODEs Using Knowledge Distillation
  13. Effect of Temperature on Knowledge Distillation
  14. Effect of the Teacher Model on Knowledge Distillation
  15. The Robustness of Neural ODEs against Adversarial Examples
  16. ...and 4 more sections

Figures (1)

  • Figure 1: The classification accuracy of Neural ODEs trained by knowledge distillation on CIFAR10 and SVHN under three white-box attacks.