Exploiting Object-based and Segmentation-based Semantic Features for Deep Learning-based Indoor Scene Classification

TL;DR

This work tackles indoor scene classification by addressing intra-category variation and inter-category ambiguity through richer semantic representations. It introduces Segmentation-based Hu-Moments Features (SHMFs) and a three-branch network, GOS$^2$F$^2$App, that fuses global RGB features with object-based cues (SFV, SFM) and segmentation-based cues (SHMFs, SSFs) derived from DeepLabv3+ and YOLOv3. The SHMFs provide a Hu-moments-based shape characterization of segmentation categories, which, when combined with segmentation-based and object-based features, yields a more discriminative representation. On SUN RGB-D and NYUv2, the method achieves state-of-the-art results ($63.7\%$ and $80.1\%$ respectively), with ablations showing the strong benefit of aggregating multiple semantic sources and the impact of segmentation-mask quality on performance. This approach highlights the practical value of integrating semantic cues for robust indoor-scene understanding and points to future work in attention-based feature selection and graph-based semantic modeling.

Abstract

Indoor scenes are usually characterized by scattered objects and their relationships, which turns the indoor scene classification task into a challenging computer vision task. Despite the significant performance boost in classification tasks achieved in recent years, provided by the use of deep-learning-based methods, limitations such as inter-category ambiguity and intra-category variation have been holding back their performance. To overcome such issues, gathering semantic information has been shown to be a promising source of information towards a more complete and discriminative feature representation of indoor scenes. Therefore, the work described in this paper uses both semantic information, obtained from object detection, and semantic segmentation techniques. While object detection techniques provide the 2D location of objects allowing to obtain spatial distributions between objects, semantic segmentation techniques provide pixel-level information that allows to obtain, at a pixel-level, a spatial distribution and shape-related features of the segmentation categories. Hence, a novel approach that uses a semantic segmentation mask to provide Hu-moments-based segmentation categories' shape characterization, designated by Segmentation-based Hu-Moments Features (SHMFs), is proposed. Moreover, a three-main-branch network, designated by GOS$^2$F$^2$App, that exploits deep-learning-based global features, object-based features, and semantic segmentation-based features is also proposed. GOS$^2$F$^2$App was evaluated in two indoor scene benchmark datasets: SUN RGB-D and NYU Depth V2, where, to the best of our knowledge, state-of-the-art results were achieved on both datasets, which present evidences of the effectiveness of the proposed approach.

Figures (6)

• Figure 1: Overview of the proposed GOS$^2$F$^2$App, which consists of three main branches. The object-based branch (top branch) generates object bounding boxes, from which the SFV and SFM object-based features are extracted. These features are further exploited by the SVF-CNN and SFM-CNN, respectively. In the segmentation branch (middle branch), a semantic segmentation mask is generated, from which the SHFMs and SSFs segmentation-based features are extracted. These features are later exploited by the SHFM-CNN and SSF-CNN, respectively. The global branch (bottom branch) employs a state-of-the-art deep-learning-based feature extraction approach to learn global features. All the output features converge to a feature fusion stage to obtain a scene category prediction.
• Figure 2: Example of Hu-moments features obtained in different objects' shape conditions.
• Figure 3: Two-step learning of the GOS$^2$F$^2$App. In the first step, the global branch parameters, composed of the backbone network and an FC layer, are trained. The backbone network parameters are initialized using the ImageNet pre-trained model. In the second learning step, the global branch uses the parameters trained in the first step, and only the remaining parameters (segmentation branch, object-based branch and feature fusion parameters) are trained.
• Figure 4: Confusion matrix of GOS$^2$F$^2$App on the SUN RGB-D dataset (Classes: 0 = bathroom, 1 = bedroom, 2 = classroom, 3 = computer room, 4 = conference room, 5 = corridor, 6 = dining area, 7 = dining room, 8 = discussion area, 9 = furniture store, 10 = home office, 11 = kitchen, 12 = lab, 13 = lecture theater , 14 = library, 15 = living room, 16 = office, 17 = rest space, 18 = study space).
• Figure 5: Misclassification examples of the GOS$^2$F$^2$App on the SUN RGB-D dataset. For each scene category, an RGB image is presented. Also, to present the high level of similarity between scene categories, an RGB image with object bounding boxes overlaid on detected objects and an RGB image with a segmentation mask generated by the S75 model superimposed, are presented.