The Impact of Background Removal on Performance of Neural Networks for Fashion Image Classification and Segmentation

TL;DR

This work investigates whether removing backgrounds from fashion images improves neural network performance. By conducting a controlled set of experiments across model architectures, initializations, data augmentations, and tasks using FashionStyle14 for classification and Fashionpedia for segmentation, it shows that background removal can boost $Top-1$ accuracy by up to about $5\%$ for simple, from-scratch classifiers, but harms deeper networks, regularization, and transfer learning with pre-trained weights. The findings indicate minimal or negative impact on instance and semantic segmentation, largely due to loss of contextual background information and incompatibilities with common data-augmentation tricks. Practically, background removal is not a universal preprocessing fix for fashion tasks; its benefits are limited to shallow classification scenarios, and its use should be weighed against potential overfitting and reduced regularization in modern deep models.

Abstract

Fashion understanding is a hot topic in computer vision, with many applications having great business value in the market. Fashion understanding remains a difficult challenge for computer vision due to the immense diversity of garments and various scenes and backgrounds. In this work, we try removing the background from fashion images to boost data quality and increase model performance. Having fashion images of evident persons in fully visible garments, we can utilize Salient Object Detection to achieve the background removal of fashion data to our expectations. A fashion image with the background removed is claimed as the "rembg" image, contrasting with the original one in the fashion dataset. We conducted extensive comparative experiments with these two types of images on multiple aspects of model training, including model architectures, model initialization, compatibility with other training tricks and data augmentations, and target task types. Our experiments show that background removal can effectively work for fashion data in simple and shallow networks that are not susceptible to overfitting. It can improve model accuracy by up to 5% in the classification on the FashionStyle14 dataset when training models from scratch. However, background removal does not perform well in deep neural networks due to incompatibility with other regularization techniques like batch normalization, pre-trained initialization, and data augmentations introducing randomness. The loss of background pixels invalidates many existing training tricks in the model training, adding the risk of overfitting for deep models.

Figures (9)

• Figure 1: Original image samples from FashionStyle14 takagi_what_2017 and Fashionpedia jia_fashionpedia_2020.
• Figure 2: Rembg image samples from FashionStyle14 takagi_what_2017 and Fashionepdia jia_fashionpedia_2020.
• Figure 3: The general pipeline of comparative experiments.
• Figure 4: The pipeline for image classification experiments on FashionStyle14.
• Figure 5: The performance of the trained from scratch models on rembg images with removed backgrounds compared to the original FashionStyle14 images.