Region-based Content Enhancement for Efficient Video Analytics at the Edge

TL;DR

Region-based content enhancement tackles the cost of applying heavy super-resolution to entire frames by focusing augmentation on high-impact regions. RegenHance introduces three synergistic components—MB-based region importance prediction, region-aware enhancement with cross-stream MB selection and 2D region packing, and profile-based execution planning—to achieve high accuracy and throughput on edge devices. Across object detection and semantic segmentation tasks, RegenHance yields 10–19% accuracy gains and 2–3x end-to-end throughput versus frame-based enhancement baselines, supported by extensive evaluation on five diverse devices. The approach enables scalable, real-time video analytics at the edge, with robust performance across workloads, models, and resolutions, and offers actionable strategies for region-level processing and resource scheduling in constrained environments.

Abstract

Video analytics is widespread in various applications serving our society. Recent advances of content enhancement in video analytics offer significant benefits for the bandwidth saving and accuracy improvement. However, existing content-enhanced video analytics systems are excessively computationally expensive and provide extremely low throughput. In this paper, we present region-based content enhancement, that enhances only the important regions in videos, to improve analytical accuracy. Our system, RegenHance, enables high-accuracy and high-throughput video analytics at the edge by 1) a macroblock-based region importance predictor that identifies the important regions fast and precisely, 2) a region-aware enhancer that stitches sparsely distributed regions into dense tensors and enhances them efficiently, and 3) a profile-based execution planer that allocates appropriate resources for enhancement and analytics components. We prototype RegenHance on five heterogeneous edge devices. Experiments on two analytical tasks reveal that region-based enhancement improves the overall accuracy of 10-19% and achieves 2-3x throughput compared to the state-of-the-art frame-based enhancement methods.

Figures (35)

• Figure 1: Performance of the frame-based methods. The state-of-the-art selective SR provides poor E2E throughput.
• Figure 2: An example eregion worth to be enhanced for object detection (OD) bounding with a rectangle box.
• Figure 3: Distribution of eregions in various videos. In large amounts of frames, eregions occupy only a small portion.
• Figure 4: Latency of enhancement. The same H*W (e.g., 64*64) input, no matter pixel values, yields the same latency.
• Figure 5: Region-based enhancement saves remarkable (2.4$\times$) latency, but prior region selection methods themselves (e.g., RoI detection du2020server) produces too high computing cost.