Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RBF Weighted Hyper-Involution for RGB-D Object Detection

Mehfuz A Rahman, Khushal Das, Jiju Poovvancheri, Neil London, Dong Chen

TL;DR

A real-time two-stream RGBD object detection model that introduces a dynamic radial basis function (RBF) weighted depth-based hyper-involution that adjusts dynamically based on spatial interaction patterns in raw depth maps, and an up-sampling based trainable fusion layer that combines extracted depth and color image features without obstructing information transfer between them.

Abstract

A vast majority of augmented reality devices come equipped with depth and color cameras. Despite their advantages, extracting both photometric and depth features simultaneously in real-time remains challenging due to inherent differences between depth and color images. Furthermore, standard convolution operations are insufficient for extracting information directly from raw depth images, leading to inefficient intermediate representations. To address these issues, we propose a real-time two-stream RGBD object detection model. Our model introduces two new components: a dynamic radial basis function (RBF) weighted depth-based hyper-involution that adjusts dynamically based on spatial interaction patterns in raw depth maps, and an up-sampling based trainable fusion layer that combines extracted depth and color image features without obstructing information transfer between them. Experimental results demonstrate that the proposed approach achieves the strongest performance among existing RGB-D 2D object detection methods on NYU Depth V2, while remaining competitive on the SUN RGB-D benchmark.

RBF Weighted Hyper-Involution for RGB-D Object Detection

TL;DR

A real-time two-stream RGBD object detection model that introduces a dynamic radial basis function (RBF) weighted depth-based hyper-involution that adjusts dynamically based on spatial interaction patterns in raw depth maps, and an up-sampling based trainable fusion layer that combines extracted depth and color image features without obstructing information transfer between them.

Abstract

A vast majority of augmented reality devices come equipped with depth and color cameras. Despite their advantages, extracting both photometric and depth features simultaneously in real-time remains challenging due to inherent differences between depth and color images. Furthermore, standard convolution operations are insufficient for extracting information directly from raw depth images, leading to inefficient intermediate representations. To address these issues, we propose a real-time two-stream RGBD object detection model. Our model introduces two new components: a dynamic radial basis function (RBF) weighted depth-based hyper-involution that adjusts dynamically based on spatial interaction patterns in raw depth maps, and an up-sampling based trainable fusion layer that combines extracted depth and color image features without obstructing information transfer between them. Experimental results demonstrate that the proposed approach achieves the strongest performance among existing RGB-D 2D object detection methods on NYU Depth V2, while remaining competitive on the SUN RGB-D benchmark.
Paper Structure (21 sections, 4 equations, 15 figures, 9 tables)

This paper contains 21 sections, 4 equations, 15 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Detection based on HHA Formats
  4. Detection using Raw Depth Maps
  5. Alternatives to Standard Convolution
  6. The Model
  7. Two Streams Architecture
  8. Depth Aware Hyper-involution
  9. Fusion Stage
  10. Implementation Details
  11. Experiments
  12. Performance on Benchmark Datasets
  13. Outdoor RGB-D Dataset
  14. Performance on Outdoor RGB-D Dataset
  15. Feature Maps Analysis
  16. ...and 6 more sections

Figures (15)

  • Figure 1: Few instances where the usefulness of depth for object detection is visible. Image courtesy: Silberman:ECCV12polseno_2020ranftl2021visionrankuzz.com_2020fun_dog_tiger
  • Figure 2: The proposed two streams and single stage detection architecture for real-time applications.
  • Figure 3: Depth Aware Hyper-involution: Depth similarity creates a depth-aware filter, with a hyper-network generating weights for each image region.
  • Figure 4: Difference between pixels of RGB image and its corresponding depth map.
  • Figure 5: The filter generation hyper-network learns filter weights for each RGB pixel individually
  • ...and 10 more figures