A recommender network perspective on the informational value of critics and crowds

TL;DR

It is found that critics are more consistent than amateurs, and thus their advice is more predictive than advice from amateurs, and the framework is broadly applicable and can be leveraged to devise good decision strategies and more transparent recommender systems.

Abstract

How do the ratings of critics and amateurs compare and how should they be combined? Previous research has produced mixed results about the first question, while the second remains unanswered. We have created a new, unique dataset, with wine ratings from critics and amateurs, and simulated a recommender system using the k-nearest-neighbor algorithm. We then formalized the advice seeking network spanned by that algorithm and studied people's relative influence. We find that critics are more consistent than amateurs, and thus their advice is more predictive than advice from amateurs. Getting advice from both groups can further boost performance. Our network theoretic approach allows us to identify influential critics, talented amateurs, and the information flow between groups. Our results provide evidence about the informational function of critics, while our framework is broadly applicable and can be leveraged to devise good decision strategies and more transparent recommender systems.

Figures (12)

• Figure 1: Intercorrelations with members of the same group and all other individuals.Left: The position of 14 professional critics and 120 amateurs on a 2-dimensional plane defined by mean taste similarity (i.e., mean correlation) and dispersion in taste similarity (i.e., SD of correlations) with members of the same group. Right: The position of the same 14 professional critics and 120 amateurs on the same plane, but this time with taste similarity calculated across all individuals (professional critics and amateurs). The color in both panels indicates whether an individual is a professional critic or an amateur and the point size in the right panel indicates recommender potential. The dotted orange and purple lines indicate the average correlations and dispersion in taste similarity for amateurs and critics. Only correlations when two individuals had an overlap of more than 5 ratings were considered in this graph. Initials of professional critics: WA --- Lisa Perotti Brown, NM --- Neal Martin, JR --- Jancis Robinson, TA --- Tim Atkin, B & D --- Michel Bettanne and Thierry Desseauve, JS --- James Suckling, JL --- Jeff Leve, De --- Steven Spurrier, James Lawther, Beverley Blanning and Jane Anson, RVF --- Olivier Poels, Hélène Durange, and Philippe Maurange, JA --- Jane Anson, LeP --- Jacques Dupont, PW --- Ronald DeGroot, RG --- Rene Gabriel, and CK --- Chris Kissack.
• Figure 2: Performance of the recommender system for different groups (left) and individuals (right).Left: The average performance of the k-nn algorithm based only on amateurs, only on critics, or both amateurs and critics for different values of $k$ for the amateur and critics groups. Right: The individual level performance of the k-nn algorithm based only on amateurs, only on critics, or both amateurs and critics for $k=5$.
• Figure 3: The recommender potential and recommender influence of different individuals. Nodes represent individuals, node size represents recommender potential (upper left circle) or recommender influence (bottom right circle) of different people in the recommender network spanned by the k-nn algorithm. Bottom Left: Orange edges indicate that advice is sought (or provided) from an amateur and purple edges indicate that advice is sought (or provided) from a professional critic. The colour of the nodes indicates the accuracy of the algorithm for different individuals in the dataset. Edges with weights smaller than 0.05 do not appear in the visualization to prevent overcrowding the graph. Upper Left: The The advice-seeking network produced by the initial call of k-nn, disregarding missing values (i.e., recommender potential). The edges (arrows) are pointing to the individuals from whom k-nn first seeks advice for the target individual. Lower Right: The influence graph eventually produced by k-nn. When an individual called by k-nn has not rated a wine label, the next individual in the correlation rank is consulted. This process continues until $k$ advisers have been found or until the pool of potential advisers is exhausted. Upper Right: Amateurs and professional critics placed on a 2-dimensional plane defined by the number of items they have evaluated (x-axis) and their recommender potential (y-axis). Critics are depicted with purple color and amateurs with yellow. Node size indicates the total influence of different individuals.
• Figure 4: Homophily index of critics and amateursLeft and Right: The homophily index of amateurs and critics as a function of the value $k$ in the $k$-nearest neighbors algorithm. Different $\rho$ values are represented with lines of different color. The horizontal dashed lines represent homophily baselines corresponding to the proportion of group members in the population (group weight) and the proportion of the ratings contributed by the members of each group (count weight).
• Figure 5: Intercorrelations with individuals belonging in the same group and all other individuals when the data are balanced for sparsity.Left: The position of 14 professional critics and 120 amateurs on a 2-dimensional plane defined by mean taste similarity (i.e., mean correlation) and dispersion in taste similarity (i.e., standard deviation of correlations) with members of the same group. Right: The position of the same 14 professional critics and 120 amateurs on the same plane, but this time with taste similarity calculated across all individuals (professional critics and amateurs). The color in both panels indicates whether an individual is a professional critic or an amateur and the point size in the right panel indicates the recommender potential of different individuals for $k = 5$ and $\rho=1$.