Breaking the accuracy-resource dilemma: a lightweight adaptive video inference enhancement

TL;DR

Frames the resource-accuracy trade-off in video inference and introduces a lightweight fuzzy controller FC-r that, using real-time device state and video context, enables dynamic switching between models of different scales by exploiting spatiotemporal correlations across frames. Proposes FC-r-based VI control with fuzzification of inputs $GU$, $GT$, and $NT$, a rule-based inference engine with max-aggregation, and defuzzification to yield a crisp inference score guiding model selection; the mechanism is designed for real-time adaptivity with $O(1)$ overhead. Evaluated on VisDrone and UA-DETRAC with YOLOv8 s/m/l on Jetson Orin NX and a PC; four scenarios including single-scale and adaptive inference; results show only 24 switches in 2000 frames, detection performance comparable to the large model under high target density, and reduced temperature rise and resource utilization. Conclusion: the approach enables robust, energy-efficient video inference on resource-constrained devices via context-aware, hardware-conscious model selection.

Abstract

Existing video inference (VI) enhancement methods typically aim to improve performance by scaling up model sizes and employing sophisticated network architectures. While these approaches demonstrated state-of-the-art performance, they often overlooked the trade-off of resource efficiency and inference effectiveness, leading to inefficient resource utilization and suboptimal inference performance. To address this problem, a fuzzy controller (FC-r) is developed based on key system parameters and inference-related metrics. Guided by the FC-r, a VI enhancement framework is proposed, where the spatiotemporal correlation of targets across adjacent video frames is leveraged. Given the real-time resource conditions of the target device, the framework can dynamically switch between models of varying scales during VI. Experimental results demonstrate that the proposed method effectively achieves a balance between resource utilization and inference performance.

Figures (4)

• Figure 1: Overall architecture of the proposed algorithm system, which consists of two main components. The upper section illustrates the FC responsible for real-time fuzzy inference. The lower section is divided into two parts: the left part handles VI, while the right part integrates models of different scales and complexities.
• Figure 2: Detection Results on the VisDrone and UA-DETRAC
• Figure 3: The AVTG on the Jetson and PC
• Figure 4: Curves of temperature variation with frame rate for different methods on video stream frames of the UA-DETRAC