GMC: A General Framework of Multi-stage Context Learning and Utilization for Visual Detection Tasks

TL;DR

GMC introduces a general, three-stage framework for context-aware visual detection that unifies local, semantic, and spatial context across data labeling, model training, and post-processing. By implementing Local Context Representation (LCR), Semantic Context Fusion (SCF), and Spatial Context Reasoning (SCR) with automatic task-driven configuration, GMC improves detection performance across storefront, urban pedestrian, and COCO tasks for both CNN- and transformer-based detectors. Key findings show substantial gains in mAP and recall for CNNs on storefronts, notable improvements in pedestrian detection with CityPersons+, and consistent, though dataset-dependent, gains on COCO, highlighting the framework’s versatility and practical impact. The work demonstrates that integrating contextual cues at multiple stages can yield robust improvements with minimal code changes, offering a flexible blueprint for context-enabled detection in real-world scenarios.

Abstract

Various contextual information has been employed by many approaches for visual detection tasks. However, most of the existing approaches only focus on specific context for specific tasks. In this paper, GMC, a general framework is proposed for multistage context learning and utilization, with various deep network architectures for various visual detection tasks. The GMC framework encompasses three stages: preprocessing, training, and post-processing. In the preprocessing stage, the representation of local context is enhanced by utilizing commonly used labeling standards. During the training stage, semantic context information is fused with visual information, leveraging prior knowledge from the training dataset to capture semantic relationships. In the post-processing stage, general topological relations and semantic masks for stuff are incorporated to enable spatial context reasoning between objects. The proposed framework provides a comprehensive and adaptable solution for context learning and utilization in visual detection scenarios. The framework offers flexibility with user-defined configurations and provide adaptability to diverse network architectures and visual detection tasks, offering an automated and streamlined solution that minimizes user effort and inference time in context learning and reasoning. Experimental results on the visual detection tasks, for storefront object detection, pedestrian detection and COCO object detection, demonstrate that our framework outperforms previous state-of-the-art detectors and transformer architectures. The experiments also demonstrate that three contextual learning components can not only be applied individually and in combination, but can also be applied to various network architectures, and its flexibility and effectiveness in various detection scenarios.

Figures (12)

• Figure 1: The overview of GMC, our general framework of multi-stage context learning and utilization for visual detection tasks. We design a user configuration mechanism for automating the process for various detection tasks and with different network models. Each context component is guided by user-defined parameters with minimum modification of the system when applying to different deep learning models and visual tasks.
• Figure 2: Details of our GMC framework, the general framework of multi-stage context learning and utilization for visual detection tasks. We design a user configuration mechanism for automating the process for various detection tasks (e.g., storefront object detection, pedestrian detection), using different base detectors (e.g. a CNN model Faster R-CNN (FRCNN) and a transformer model DETR. Three context learning and utilization components - (a) Local Context Representation, (b) Semantic Context Fusion, and (c) Spatial Context Reasoning, guide the deep learning models during data labeling, model training and post-processing stages. Each component can be applied individually and in combination. GT: Ground Truth. LC: Local Context. S: Subject. O: Object.
• Figure 3: An utilized local context representation. The local context calculator is guided by user-defined parameters and enhance the local context around the ground truth label of the object. GT: Ground Truth. LC: Local Context. FI: Final Input.
• Figure 4: The visualization of Semantic Context Fusion. We use category information as the semantic context cues to generate semantic spaces for visual detection tasks.
• Figure 5: The visualization of common used topological relationships from clementini1993small and egenhofer1991point.