A Perspective on Deep Vision Performance with Standard Image and Video Codecs

TL;DR

It is found that using JPEG and H.264 coding significantly deteriorates the accuracy across a broad range of vision tasks and models, and extends beyond image and action classification to localization and dense prediction tasks, thus providing a more comprehensive perspective.

Abstract

Resource-constrained hardware, such as edge devices or cell phones, often rely on cloud servers to provide the required computational resources for inference in deep vision models. However, transferring image and video data from an edge or mobile device to a cloud server requires coding to deal with network constraints. The use of standardized codecs, such as JPEG or H.264, is prevalent and required to ensure interoperability. This paper aims to examine the implications of employing standardized codecs within deep vision pipelines. We find that using JPEG and H.264 coding significantly deteriorates the accuracy across a broad range of vision tasks and models. For instance, strong compression rates reduce semantic segmentation accuracy by more than 80% in mIoU. In contrast to previous findings, our analysis extends beyond image and action classification to localization and dense prediction tasks, thus providing a more comprehensive perspective.

Figures (9)

• Figure 1: Deep vision performance on standard coded images/videos. We demonstrate that using standard image and video coding can vastly deteriorate the accuracy of current deep vision models. We visualize the original (left) and coded (right) image/video and the respective models' prediction. For optical flow estimation, we overlay the first and second frame. Best viewed in color; zoom in for details.
• Figure 2: Relative semantic segmentation accuracy on JPEG-coded Cityscapes (val) dataset. The accuracy of all models vastly decreases as the compression rate increases (lower JPEG quality). ResNet-18 backbone in blue ($\blacksquare\!\!\!\!\!\blacksquare$), ResNet-50 backbone in yellow ($\blacksquare\!\!\!\!\!\blacksquare$), and ResNet-101 backbone in orange ($\blacksquare\!\!\!\!\!\blacksquare$). Best viewed in color.
• Figure 3: Qualitative semantic segmentation example on a JPEG-coded Cityscapes (val) images. As the compression rate increases (lower JPEG quality), the DeepLabV3 (w/ ResNet-18 backbone) model is not able to maintain its accuracy. For a JPEG quality of 3, the semantic segmentation accuracy completely breaks down. The mIoU relative to the uncoded baseline prediction is computed on a per-image basis. We report the JPEG file size for the coded images. Best viewed in color; zoom in for details.
• Figure 4: Relative object detection accuracy on the JPEG-coded COCO (val) dataset. The accuracy of all DETR and Faster R-CNN variants vastly deteriorates as the compression rate increases (lower JPEG quality). We report the mAP in blue ($\blacksquare\!\!\!\!\!\blacksquare$), the mAP50 in yellow ($\blacksquare\!\!\!\!\!\blacksquare$), and the mAP75 in orange ($\blacksquare\!\!\!\!\!\blacksquare$). Best viewed in color.
• Figure 5: Relative image classification accuracy on the JPEG-coded ImageNet-1k (val) dataset. The relative accuracy (i.e., w.r.t. the pseudo labels) of all models vastly deteriorates as the compression rate increases (lower JPEG quality). We report the top-1 accuracy in blue ($\blacksquare\!\!\!\!\!\blacksquare$) and the top-5 accuracy in yellow ($\blacksquare\!\!\!\!\!\blacksquare$). Best viewed in color.