Re-Ranking News Comments by Constructiveness and Curiosity Significantly Increases Perceived Respect, Trustworthiness, and Interest

TL;DR

The paper investigates whether re-ranking news comments by prosocial attributes can foster healthier online discussions. It uses a rich-media, 4x2 between-subjects design to compare Control, Constructiveness, Curiosity, and Personal Stories rankings across Politics and Dining article contexts with 460 participants. The results show that re-ranking by constructiveness and curiosity improves perceived respect, trustworthiness, and informativeness in the politics article, while effects vary by article type, with the dining context yielding informativeness gains under certain rankings. The findings suggest that incorporating prosocial speech traits into ranking could promote healthier, less polarized dialogue without sacrificing engagement, though context-aware design and tradeoffs such as message length must be considered.

Abstract

Online commenting platforms have commonly developed systems to address online harms by removing and down-ranking content. An alternative, under-explored approach is to focus on up-ranking content to proactively prioritize prosocial commentary and set better conversational norms. We present a study with 460 English-speaking US-based news readers to understand the effects of re-ranking comments by constructiveness, curiosity, and personal stories on a variety of outcomes related to willingness to participate and engage, as well as perceived credibility and polarization in a comment section. In our rich-media survey experiment, participants across these four ranking conditions and a control group reviewed prototypes of comment sections of a Politics op-ed and Dining article. We found that outcomes varied significantly by article type. Up-ranking curiosity and constructiveness improved a number of measures for the Politics article, including perceived Respect, Trustworthiness, and Interestingness of the comment section. Constructiveness also increased perceptions that the comments were favorable to Republicans, with no condition worsening perceptions of partisans. Additionally, in the Dining article, personal stories and constructiveness rankings significantly improved the perceived informativeness of the comments. Overall, these findings indicate that incorporating prosocial qualities of speech into ranking could be a promising approach to promote healthier, less polarized dialogue in online comment sections.

Figures (7)

• Figure 1: On the left, diagram depicting the breakdown of experimental conditions for participants. The comments from condition A. were taken directly from the displayed order on nytimes.com site as a control. The comments for conditions B., C., and D. were ordered by the top-scored comments for each respective classifier, excluding replies. On the right, example of an interactive prototype accessed by participants in condition A. for the politics article. User profile images, engagement data, New York Times-specific branding and features were removed. User names were randomly generated, with names consistent between conditions.
• Figure 2: The means of Hostility (a), Respect (b), Informativeness (c), Trustworthiness (d), Interestingness (e), and Future Visit (d) in the dining article (blue), and politics article (red) for each ranking condition, shown with 95% Confidence Interval error bars.
• Figure 3: The means of Democrat Favorability (a) and Republican Favorability (b) in the comment section by each ranking condition, on a scale from "Very Unfavorable" (1) to "Very Favorable" (5). Question wording: "In the previous comment section, how would you rate the commenters' attitudes towards [Republican or Democratic] voters?"
• Figure 4: The model card for constructiveness, curiosity, and personal stories.
• Figure 5: Two-way ANOVA Tests of Between-Subjects Effects