Context-aware Multi-Model Object Detection for Diversely Heterogeneous Compute Systems

TL;DR

The paper tackles the inefficiency of deploying a single DNN for continuous object detection on energy-constrained autonomous systems. It proposes SHIFT, a context-aware, multi-model, multi-accelerator framework composed of offline ODM trait characterization, a fast context-driven SHIFT Scheduler, and a Dynamic Model Loader to manage memory and model swaps. The key contributions are the confidence-graph-based accuracy prediction, the low-overhead runtime scheduler, and the memory-aware dynamic loader, all demonstrated on heterogeneous hardware with substantial energy and latency benefits (up to $7.5\times$ energy and $2.8\times$ latency reductions) while preserving competitive detection quality measured by $IoU$. This approach enables real-time, energy-efficient OD in diverse compute environments, with practical impact on autonomous systems that must balance accuracy, speed, and power consumption in dynamic scenarios.

Abstract

In recent years, deep neural networks (DNNs) have gained widespread adoption for continuous mobile object detection (OD) tasks, particularly in autonomous systems. However, a prevalent issue in their deployment is the one-size-fits-all approach, where a single DNN is used, resulting in inefficient utilization of computational resources. This inefficiency is particularly detrimental in energy-constrained systems, as it degrades overall system efficiency. We identify that, the contextual information embedded in the input data stream (e.g. the frames in the camera feed that the OD models are run on) could be exploited to allow a more efficient multi-model-based OD process. In this paper, we propose SHIFT which continuously selects from a variety of DNN-based OD models depending on the dynamically changing contextual information and computational constraints. During this selection, SHIFT uniquely considers multi-accelerator execution to better optimize the energy-efficiency while satisfying the latency constraints. Our proposed methodology results in improvements of up to 7.5x in energy usage and 2.8x in latency compared to state-of-the-art GPU-based single model OD approaches.

Figures (6)

• Figure 1: Comparison of (a) single-model with multiple parameter sizes on the left against (b) multi-model object detection architectures on the right. The larger the value along each axis the better: a perfect model would be largest triangle across all axes.
• Figure 2: Single model object detection efficiency on GPU for commonly used DNNs and their variations on a test set for continuous detection and tracking of an aerial drone. Efficiency is quantified by intersection over union (IoU) per Joule of energy (see Sec. \ref{['chap:experiments']} for details).
• Figure 3: Scenario 1: Drone navigates across multiple backgrounds at varying distances from the camera.
• Figure 4: Scenario 2: Drone navigates across multiple backgrounds at a fixed distance.
• Figure 5: Sensitivity analysis of the SHIFT parameters against the mean accuracy, energy, and latency values.