Seam Carving as Feature Pooling in CNN

TL;DR

This paper addresses whether seam carving can replace max pooling in CNNs for image classification by integrating a content-aware pooling step into a shallow two-convolution network and evaluating on a two-class subset of the Caltech-UCSD Birds dataset. The model uses seam carving after convolution to reduce feature maps, with inputs of $32\times 32$ and intermediate feature maps of sizes up to $16\times 16$. Results show seam carving yields higher accuracy ($0.80$ vs $0.70$) and lower evaluation loss ($0.472$ vs $0.588$), with a slight F1 improvement and visual evidence of better structural information preservation in feature maps. The work demonstrates the potential of content-aware pooling in CNNs but notes the need for broader validation, hybrid pooling strategies, and exploration across diverse datasets and modern training techniques.

Abstract

This work investigates the potential of seam carving as a feature pooling technique within Convolutional Neural Networks (CNNs) for image classification tasks. We propose replacing the traditional max pooling layer with a seam carving operation. Our experiments on the Caltech-UCSD Birds 200-2011 dataset demonstrate that the seam carving-based CNN achieves better performance compared to the model utilizing max pooling, based on metrics such as accuracy, precision, recall, and F1-score. We further analyze the behavior of both approaches through feature map visualizations, suggesting that seam carving might preserve more structural information during the pooling process. Additionally, we discuss the limitations of our approach and propose potential future directions for research.

Figures (8)

• Figure 1: Example of Seam Carving, Max Pooling and Resizing (downsampling).
• Figure 2: Examples illustration of the methodolgies of seam carving and max pooling. For seam carving, the energy maps is shown. And for max pooling, the corresponding windows are shown with bold border. For both, the orange items in the grids are selected to be removed during the processes.
• Figure 3: Integration of seam carving as feature pooling in CNN. (a) shows a basic CNN architecture blocks, indicating the use of seam carving layer. (b) shows an example CNN PyTorch model with seam carving function.
• Figure 4: Few examples from the classes: Bobolink (Class - 0) and Indigo Bunting (Class - 1).
• Figure 5: PyTorch model summary.