Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ABO: Abandon Bayer Filter for Adaptive Edge Offloading in Responsive Augmented Reality

Yongxuan Han, Shengzhong Liu, Fan Wu, Guihai Chen

TL;DR

AR demands real-time, accurate DNN analytics under dynamic bandwidth. ABO tackles this by decoupling demosaicing from offloading and introducing a tile-based RAW image neural codec paired with a bandwidth-aware dynamic controller. The approach uses an asymmetric autoencoder with multiple configurations, preserves CFA information, and employs content-aware tile selection and offline profiling to optimize accuracy-bandwidth tradeoffs. Prototyped on a real AR hardware pipeline, ABO achieves up to 15% higher downstream accuracy, over 40% greater frame throughput, and around 30% lower end-to-end latency than state-of-the-art baselines, while remaining robust in low-light and high-motion conditions.

Abstract

Bayer-patterned color filter array (CFA) has been the go-to solution for color image sensors. In augmented reality (AR), although color interpolation (i.e., demosaicing) of pre-demosaic RAW images facilitates a user-friendly rendering, it creates no benefits in offloaded DNN analytics but increases the image channels by 3 times inducing higher transmission overheads. The potential optimization in frame preprocessing of DNN offloading is yet to be investigated. To that end, we propose ABO, an adaptive RAW frame offloading framework that parallelizes demosaicing with DNN computation. Its contributions are three-fold: First, we design a configurable tile-wise RAW image neural codec to compress frame sizes while sustaining downstream DNN accuracy under bandwidth constraints. Second, based on content-aware tiles-in-frame selection and runtime bandwidth estimation, a dynamic transmission controller adaptively calibrates codec configurations to maximize the DNN accuracy. Third, we further optimize the system pipelining to achieve lower end-to-end frame processing latency and higher throughput. Through extensive evaluations on a prototype platform, ABO consistently achieves 40% more frame processing throughput and 30% less end-to-end latency while improving the DNN accuracy by up to 15% than SOTA baselines. It also exhibits improved robustness against dim lighting and motion blur situations.

ABO: Abandon Bayer Filter for Adaptive Edge Offloading in Responsive Augmented Reality

TL;DR

AR demands real-time, accurate DNN analytics under dynamic bandwidth. ABO tackles this by decoupling demosaicing from offloading and introducing a tile-based RAW image neural codec paired with a bandwidth-aware dynamic controller. The approach uses an asymmetric autoencoder with multiple configurations, preserves CFA information, and employs content-aware tile selection and offline profiling to optimize accuracy-bandwidth tradeoffs. Prototyped on a real AR hardware pipeline, ABO achieves up to 15% higher downstream accuracy, over 40% greater frame throughput, and around 30% lower end-to-end latency than state-of-the-art baselines, while remaining robust in low-light and high-motion conditions.

Abstract

Bayer-patterned color filter array (CFA) has been the go-to solution for color image sensors. In augmented reality (AR), although color interpolation (i.e., demosaicing) of pre-demosaic RAW images facilitates a user-friendly rendering, it creates no benefits in offloaded DNN analytics but increases the image channels by 3 times inducing higher transmission overheads. The potential optimization in frame preprocessing of DNN offloading is yet to be investigated. To that end, we propose ABO, an adaptive RAW frame offloading framework that parallelizes demosaicing with DNN computation. Its contributions are three-fold: First, we design a configurable tile-wise RAW image neural codec to compress frame sizes while sustaining downstream DNN accuracy under bandwidth constraints. Second, based on content-aware tiles-in-frame selection and runtime bandwidth estimation, a dynamic transmission controller adaptively calibrates codec configurations to maximize the DNN accuracy. Third, we further optimize the system pipelining to achieve lower end-to-end frame processing latency and higher throughput. Through extensive evaluations on a prototype platform, ABO consistently achieves 40% more frame processing throughput and 30% less end-to-end latency while improving the DNN accuracy by up to 15% than SOTA baselines. It also exhibits improved robustness against dim lighting and motion blur situations.
Paper Structure (46 sections, 6 equations, 24 figures, 2 tables, 1 algorithm)

This paper contains 46 sections, 6 equations, 24 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivations
  3. DNN Offloading for Augmented Reality
  4. Demosaicing in Digital Image Processing
  5. Pre-Demosaic Offloading Savings
  6. ABO Framework
  7. Overview
  8. Tile-wise RAW Image Neural Codec
  9. Asymmetric Configurable Autoencoder.
  10. Frame Color Preservation
  11. Tile Partitioning.
  12. Learning Objectives.
  13. Dynamic Transmission Controller
  14. Adaptation Problem Formulation
  15. Offline Profiling
  16. ...and 31 more sections

Figures (24)

  • Figure 1: Comparing two image transmission types.
  • Figure 2: Typical on-sensor image processing pipeline.
  • Figure 3: Comparison of the original RAW image, ABO-noDistill decoded image and ABO-Distill decoded image with edge clearness values (Definitions given in Appendix \ref{['apdx:clearness']}). All images are demosaiced to RGB formats for better viewing.
  • Figure 4: Frame processing time comparison between serialized procedure in JPEG and pipelined procedure in ABO.
  • Figure 5: ABO framework overview.
  • ...and 19 more figures