Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Resource Efficient Perception for Vision Systems

A V Subramanyam, Niyati Singal, Vinay K Verma

TL;DR

The paper tackles the challenge of processing ultra-high-resolution images under limited GPU memory by introducing a unified patch-based perception framework that integrates local patch features with a global context via a downsampled full-image patch. A base extractor and a lightweight aggregator cooperate to produce a latent representation $Z$, with memory-efficient inner iterations and gradient accumulation enabling selective patch updates while preserving global information. The approach extends PatchGD to image classification, object detection, and segmentation, introducing task-specific mechanisms for patch sampling, global fusion, and compact latent representations. Empirical results across seven benchmarks demonstrate competitive performance and the ability to train on resource-constrained devices such as the Jetson Nano, highlighting practical implications for edge computing and large-scale high-resolution vision tasks.

Abstract

Despite the rapid advancement in the field of image recognition, the processing of high-resolution imagery remains a computational challenge. However, this processing is pivotal for extracting detailed object insights in areas ranging from autonomous vehicle navigation to medical imaging analyses. Our study introduces a framework aimed at mitigating these challenges by leveraging memory efficient patch based processing for high resolution images. It incorporates a global context representation alongside local patch information, enabling a comprehensive understanding of the image content. In contrast to traditional training methods which are limited by memory constraints, our method enables training of ultra high resolution images. We demonstrate the effectiveness of our method through superior performance on 7 different benchmarks across classification, object detection, and segmentation. Notably, the proposed method achieves strong performance even on resource-constrained devices like Jetson Nano. Our code is available at https://github.com/Visual-Conception-Group/Localized-Perception-Constrained-Vision-Systems.

Resource Efficient Perception for Vision Systems

TL;DR

The paper tackles the challenge of processing ultra-high-resolution images under limited GPU memory by introducing a unified patch-based perception framework that integrates local patch features with a global context via a downsampled full-image patch. A base extractor and a lightweight aggregator cooperate to produce a latent representation , with memory-efficient inner iterations and gradient accumulation enabling selective patch updates while preserving global information. The approach extends PatchGD to image classification, object detection, and segmentation, introducing task-specific mechanisms for patch sampling, global fusion, and compact latent representations. Empirical results across seven benchmarks demonstrate competitive performance and the ability to train on resource-constrained devices such as the Jetson Nano, highlighting practical implications for edge computing and large-scale high-resolution vision tasks.

Abstract

Despite the rapid advancement in the field of image recognition, the processing of high-resolution imagery remains a computational challenge. However, this processing is pivotal for extracting detailed object insights in areas ranging from autonomous vehicle navigation to medical imaging analyses. Our study introduces a framework aimed at mitigating these challenges by leveraging memory efficient patch based processing for high resolution images. It incorporates a global context representation alongside local patch information, enabling a comprehensive understanding of the image content. In contrast to traditional training methods which are limited by memory constraints, our method enables training of ultra high resolution images. We demonstrate the effectiveness of our method through superior performance on 7 different benchmarks across classification, object detection, and segmentation. Notably, the proposed method achieves strong performance even on resource-constrained devices like Jetson Nano. Our code is available at https://github.com/Visual-Conception-Group/Localized-Perception-Constrained-Vision-Systems.
Paper Structure (11 sections, 5 figures, 8 tables)

This paper contains 11 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Proposed Methodology
  4. Image Classification
  5. Object Detection
  6. Segmentation
  7. Experiments
  8. Image Classification
  9. Object Detection
  10. Segmentation
  11. Conclusion

Figures (5)

  • Figure 1: Overview of our framework. It tackles the memory limitations encountered in traditional methods by processing the image in smaller patches. A base model extracts features from each patch and from resized full image. These features are then combined into a latent representation, which essentially captures the essence of the entire image. This latent representation is then processed by an aggregator network to extract both global and fine grained details.
  • Figure 3: Predicted bounding boxes for COCO synthesized images. Resolution is 4096 $\times$ 4096.
  • Figure 4: Predicted bounding boxes on SODA-D. Resolution is 4096 $\times$ 4096
  • Figure 5: Segmented masks for DRIVE image using different Types of Unet. For better clarity, we show only a crop of image.
  • Figure : (a) Classification.