Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessment

TL;DR

Charm addresses information loss in Vision Transformer-based image aesthetic assessment by introducing a two-scale tokenization that preserves aspect ratio, high-resolution regions, and multi-scale context. It adapts pre-trained ViT position embeddings and adds learnable scale embeddings, enabling fine-tuning without extra pretraining and achieving substantial accuracy gains with lightweight backbones. Across multiple IAA and IQA datasets, Charm delivers up to 8.1% SRCC, 7.5% PLCC, and 14.8% ACC improvements while reducing computational costs, illustrating strong practicality for resource-constrained scenarios. The approach is architecture-agnostic and acts as a plug-in to enhance ViT performance, with potential for further gains through extended patch strategies and multimodal integration.

Abstract

The capacity of Vision transformers (ViTs) to handle variable-sized inputs is often constrained by computational complexity and batch processing limitations. Consequently, ViTs are typically trained on small, fixed-size images obtained through downscaling or cropping. While reducing computational burden, these methods result in significant information loss, negatively affecting tasks like image aesthetic assessment. We introduce Charm, a novel tokenization approach that preserves Composition, High-resolution, Aspect Ratio, and Multi-scale information simultaneously. Charm prioritizes high-resolution details in specific regions while downscaling others, enabling shorter fixed-size input sequences for ViTs while incorporating essential information. Charm is designed to be compatible with pre-trained ViTs and their learned positional embeddings. By providing multiscale input and introducing variety to input tokens, Charm improves ViT performance and generalizability for image aesthetic assessment. We avoid cropping or changing the aspect ratio to further preserve information. Extensive experiments demonstrate significant performance improvements on various image aesthetic and quality assessment datasets (up to 8.1 %) using a lightweight ViT backbone. Code and pre-trained models are available at https://github.com/FBehrad/Charm.

Figures (15)

• Figure 1: Traditional image preprocessing methods (right) often distort image aesthetics by rescaling and cropping images to a fixed size. To preserve critical aesthetic information, our approach (left) selectively maintains some regions in their original resolution while downscaling the others.
• Figure 2: Our approach involves partitioning the input image into fixed-size patches and selectively preserving a subset at its original resolution while downscaling others. Position and scale embeddings are then prepared to encode spatial relationships across different scales. The remaining steps follow the standard ViT. + indicates element-wise addition.
• Figure 3: An image with its aesthetic score distribution and mean aesthetic score, from the AVA dataset.
• Figure 4: Comparison of state-of-the-art IAA models' performance on the AVA dataset vs. their number of parameters. Our approach achieves comparable performance without using additional modalities and with significantly fewer parameters. To account for the significant differences in the number of parameters, we use a logarithmic scale on the x-axis. For a detailed comparison of Dinov2-small + Charm with state-of-the-art models on all datasets, see Appendix 17.
• Figure 5: The epoch at which each model achieves its highest validation performance across different datasets. Charm generally leads to faster convergence. The average number of epochs over 5 runs is reported for SPAQ and KonIQ10k.