Towards Explainable Personalized Recommendations by Learning from Users' Photos

TL;DR

The paper tackles explainable personalization by predicting the photos users would upload to justify their ratings, framing explanations as learnable recommendations. It introduces ELVis, which learns the joint probability $\Pr(\vec{u}, \vec{f})$ using a CNN-based image encoder and user embeddings to rank candidate photos as personalized explanations, selecting $\vec{f}^* = \arg\max_{\vec{f}} \Pr(\vec{u}, \vec{f})$. Using TripAdvisor restaurant reviews from six cities, the authors demonstrate that ELVis outperforms baselines in top-$n$ ranking and percentile-based metrics, capturing both individual tastes and group-level preferences. The work contributes a formal framework, a real-world dataset, and a practical method for surfacing the most attractive photos to explain recommendations, with implications for trust, transparency, and business insights into customer-highlighted product aspects.

Abstract

Explaining the output of a complex system, such as a Recommender System (RS), is becoming of utmost importance for both users and companies. In this paper we explore the idea that personalized explanations can be learned as recommendation themselves. There are plenty of online services where users can upload some photos, in addition to rating items. We assume that users take these photos to reinforce or justify their opinions about the items. For this reason we try to predict what photo a user would take of an item, because that image is the argument that can best convince her of the qualities of the item. In this sense, an RS can explain its results and, therefore, increase its reliability. Furthermore, once we have a model to predict attractive images for users, we can estimate their distribution. Thus, the companies acquire a vivid knowledge about the aspects that the clients highlight of their products. The paper includes a formal framework that estimates the authorship probability for a given pair (user, photo). To illustrate the proposal, we use data gathered from TripAdvisor containing the reviews (with photos) of restaurants in six cities of different sizes.

Figures (8)

• Figure 1: Reviews of all items are downloaded, and next filtered to get only those reviews with users' photos. For the sake of simplicity, if any user has more than one review for the same item, only the most updated one is maintained. As a result, we have a set of users, each one with $N$ reviews corresponding to $N$ different items. In order to split these data into training and test sets, $1$ review is used for testing and the rest ($N-1$) are used for training. Training set: the positive samples correspond to all the photos of the $N-1$ reviews, each one repeated 20 times (oversampling); while the negative samples are added by selecting photos taken by other users, 10 from the same item and 10 from other items. Test set: the positive samples correspond to all the photos from the review, while the negative samples are added by selecting all the photos of the same item in the training set.
• Figure 2: Topology of the network employed in ELVis to learn $\Pr(\vec{u}, \vec{f})$ from labeled pairs of users and photos $(\vec{u}, \vec{f})$
• Figure 3: Distribution of data downloaded from TripAdvisor (Gijón, Barcelona and Madrid).
• Figure 4: Distribution of data downloaded from TripAdvisor (New York, Paris and London).
• Figure 5: The first row shows the eight photos in the training set taken by a user u⃗, while the bottom box exhibits the photos of a restaurant of the test set ordered (left to right) by the joint probability estimated by ELVis. We observe that the user really likes photos of the area but not of food, and those are the pictures placed in the first positions. The seventh photo was really taken by the user in this test restaurant. Note that some photos were slightly retouched in size and/or luminosity for a correct visualization in this paper