Temporal Decisions: Leveraging Temporal Correlation for Efficient Decisions in Early Exit Neural Networks

TL;DR

This paper introduces Difference Detection and Temporal Patience as decision mechanisms for Early Exit Neural Networks and evaluates their effectiveness in health monitoring, image classification, and wake-word detection tasks.

Abstract

Deep Learning is becoming increasingly relevant in Embedded and Internet-of-things applications. However, deploying models on embedded devices poses a challenge due to their resource limitations. This can impact the model's inference accuracy and latency. One potential solution are Early Exit Neural Networks, which adjust model depth dynamically through additional classifiers attached between their hidden layers. However, the real-time termination decision mechanism is critical for the system's efficiency, latency, and sustained accuracy. This paper introduces Difference Detection and Temporal Patience as decision mechanisms for Early Exit Neural Networks. They leverage the temporal correlation present in sensor data streams to efficiently terminate the inference. We evaluate their effectiveness in health monitoring, image classification, and wake-word detection tasks. Our novel contributions were able to reduce the computational footprint compared to established decision mechanisms significantly while maintaining higher accuracy scores. We achieved a reduction of mean operations per inference by up to 80% while maintaining accuracy levels within 5% of the original model. These findings highlight the importance of considering temporal correlation in sensor data to improve the termination decision.

Figures (6)

• Figure 1: An Early Exit Neural Network with two Early Exit branches. Each branch contains an output that produces a classification vector of the same shape as the final classifier, as all outputs are intended to solve the same task.
• Figure 2: Scenes are detected by calculating the distance of an EE classifier's output to a reference predecessor. Distances above the preset threshold indicate the start of a new scene, otherwise the current scene continues.
• Figure 3: Mean operations per inference vs. accuracy on the PTB-XL test set for the different decision methods across different threshold configurations relative to the single exit version of the model.
• Figure 4: Mean operations per inference vs. accuracy on the augmented CIFAR-10 test set for the different decision methods across different threshold configurations relative to the performance of the single exit version of the model.
• Figure 5: Comparison of accuracy between the Difference Detection and Temporal Patience mechanisms and sample termination rates at each classifier based on decision threshold for speech command detection.