Personalised aesthetics with residual adapters

TL;DR

This work proposes a model based on residual learning that is capable of learning subjective, user- specific preferences over aesthetics in photography, while surpassing the state-of-the-art methods and keeping a limited number of user-specific parameters in the model.

Abstract

The use of computational methods to evaluate aesthetics in photography has gained interest in recent years due to the popularization of convolutional neural networks and the availability of new annotated datasets. Most studies in this area have focused on designing models that do not take into account individual preferences for the prediction of the aesthetic value of pictures. We propose a model based on residual learning that is capable of learning subjective, user specific preferences over aesthetics in photography, while surpassing the state-of-the-art methods and keeping a limited number of user-specific parameters in the model. Our model can also be used for picture enhancement, and it is suitable for content-based or hybrid recommender systems in which the amount of computational resources is limited.

Figures (3)

• Figure 1: Representation of the residual adapters that were tested in this paper. Each of the $\alpha^i_j$ is a set of K $1 \times 1$ convolutional filters, where $K$ is the number of $3 \times 3$ kernels in the $C_i$ layer, and is uniquely trained for each of the 37 users $j$ in our test set. For our second set of experiments with residual adapters, the $\alpha_i$ layers are actually a series of three layers of $1 \times 1$ convolutional filters: first, a layer of $K$ upcoming feature maps which outputs $K_1$ maps. Second, a layer that receives those $K_1$ feature maps and outputs other $K_1$ maps; and finally a layer that receives $K_1$ maps and outputs $K$ maps. A weight decay of $0.005$ was used on those adapters, to avoid over-fitting and to keep the adapters' initial weights close to 0. It is worth noting that, if the parameters encoded in the adapters are all equal to 0, the network with the adapters is essentially the same as a network without adapters.
• Figure 2: Visualization of the distributions of the Spearman's $\rho$ for each of the 37 workers in the test dataset on unseen data.
• Figure 3: Original (left) and enhanced (right) images obtained with our enhancement algorithm and our modified Resnet-18 network. The two original pictures were taken by the authors of this paper using smart-phones, using the HDR configuration. All the enhanced images had a greater expected rating (around $10\%$ bigger) than their corresponding original pictures. The value of $\alpha$ was set to $0.5$ so as to make the differences more visible. It can be seen that the algorithm works for pictures taken under many different lightning conditions. Additional figures are found in our repository.