BackFlip: The Impact of Local and Global Data Augmentations on Artistic Image Aesthetic Assessment

TL;DR

This work addresses the challenge of assessing aesthetic quality in artworks by recognizing that global data augmentations can disturb composition and thus ground-truth aesthetic scores. It introduces BackFlip, a local augmentation framework that segments images with Segment Anything (SAM), inpaints the background, and applies localized transforms to the segment, thereby maintaining composition while enriching training data. Across three art datasets (BAID, JenAesthetics, TAD66K) and four neural architectures, BackFlip and related local augmentations outperform traditional global augmentations in correlation metrics, with ablation studies highlighting the roles of inpainting method, segment count, and local transforms. The results underscore the importance of composition-preserving augmentation for computational aesthetics and point to future work on parameter tuning and broader applicability in aesthetic research.

Abstract

Assessing the aesthetic quality of artistic images presents unique challenges due to the subjective nature of aesthetics and the complex visual characteristics inherent to artworks. Basic data augmentation techniques commonly applied to natural images in computer vision may not be suitable for art images in aesthetic evaluation tasks, as they can change the composition of the art images. In this paper, we explore the impact of local and global data augmentation techniques on artistic image aesthetic assessment (IAA). We introduce BackFlip, a local data augmentation technique designed specifically for artistic IAA. We evaluate the performance of BackFlip across three artistic image datasets and four neural network architectures, comparing it with the commonly used data augmentation techniques. Then, we analyze the effects of components within the BackFlip pipeline through an ablation study. Our findings demonstrate that local augmentations, such as BackFlip, tend to outperform global augmentations on artistic IAA in most cases, probably because they do not perturb the composition of the art images. These results emphasize the importance of considering both local and global augmentations in future computational aesthetics research.

Figures (5)

• Figure 1: Global (Random Crop, Horizontal Flip, Rotation) and local (BoxFlip, BackFlip) image transformations on art images from the JenAesthetics dataset amirshahi2013amirshahi2013aamirshahi2014. The local data augmentations generally preserve the global composition of the images, while introducing considerable pixel-level changes that are often less perceptible to the human eye unless if they distort perceptually important shapes and objects like faces.
• Figure 2: Visualization of the BackFlip pipeline. First, we segment regions in images and inpaint the background, and then we locally flip the selected segment to implement data augmentation.
• Figure 3: Erase + Inpaint (BackFlip without inserting a segmented image) with different inpainting methods on images from TAD66K - Art.
• Figure 4: BackFlip with different local transformations on images from TAD66K - Art.
• Figure 5: Local image transformations (rotation, horizontal and vertical flip) using BackFlip with increasing number of segments. Images from the TAD66k - Art dataset.