ALERT: Accurate Learning for Energy and Timeliness

TL;DR

ALERT tackles the problem of reliable DNN inference under dynamic latency, accuracy, and energy constraints by coordinating cross-stack adaptations of both DNN models and system resource settings. It introduces a probabilistic runtime scheduler that uses a global slow-down factor $\xi$ and a Kalman-filter to predict latency, accuracy, and energy for all $(d_i, p_j)$ configurations, enabling principled selection to meet $T_{goal}$, $E_{goal}$, and $Q_{goal}$. The approach yields substantial improvements over single-layer adaptations, achieving 13% energy savings and 27% error reductions relative to non-coordinated methods, and remains near an oracle’s performance in many cases while incurring minimal overhead. Its ability to integrate Anytime DNNs and traditional DNNs under varying run-time conditions makes it practically impactful for real-time sensing and interactive AI systems.

Abstract

An increasing number of software applications incorporate runtime Deep Neural Networks (DNNs) to process sensor data and return inference results to humans. Effective deployment of DNNs in these interactive scenarios requires meeting latency and accuracy constraints while minimizing energy, a problem exacerbated by common system dynamics. Prior approaches handle dynamics through either (1) system-oblivious DNN adaptation, which adjusts DNN latency/accuracy tradeoffs, or (2) application-oblivious system adaptation, which adjusts resources to change latency/energy tradeoffs. In contrast, this paper improves on the state-of-the-art by coordinating application- and system-level adaptation. ALERT, our runtime scheduler, uses a probabilistic model to detect environmental volatility and then simultaneously select both a DNN and a system resource configuration to meet latency, accuracy, and energy constraints. We evaluate ALERT on CPU and GPU platforms for image and speech tasks in dynamic environments. ALERT's holistic approach achieves more than 13% energy reduction, and 27% error reduction over prior approaches that adapt solely at the application or system level. Furthermore, ALERT incurs only 3% more energy consumption and 2% higher DNN-inference error than an oracle scheme with perfect application and system knowledge.

Figures (11)

• Figure 1: ALERT inference system
• Figure 2: Tradeoffs for 42 DNNs (CPU2).
• Figure 3: Tradeoffs for ResNet50 at different power settings (CPU2). (Numbers inside circles are power limit settings.)
• Figure 4: Latency variance across inputs for different tasks and hardware (Most tasks have 3 boxplots for 3 hardware platforms, CPU1-2, GPU from left to right; NLP1 has an extra boxplot for Embedded; other tasks run out of memory on Embedded; every box shows the 25th--75th percentile; points beyond the whiskers are >90th or <10th).
• Figure 5: Latency variance with co-located jobs (the memory-intensive STREAM benchmark stream co-located on Embedded, CPU1-2; GPU-intensive Backprop rodinia co-located on GPU)