Examining the Utility of Self-disclosure Types for Modeling Annotators of Social Norms

TL;DR

This work investigates how self-disclosure types about annotators influence the modeling of subjective judgments in social norms tasks. It combines a theory-based taxonomy (derived from Social Penetration Theory) with automatic clustering to predict annotator verdicts on Reddit AITA data, showing that demographic information is the most predictive self-disclosure type and that a diverse, small set of annotator disclosures yields the best performance. Contrary to prior work, a few highly similar posts outperform using many posts, and theory-based categorizations generally outperform automatic clusters. The findings offer practical guidance for perspectivist NLP and demonstrate the value of richer, structured identity representations in modeling human subjectivity for NLP tasks.

Abstract

Recent work has explored the use of personal information in the form of persona sentences or self-disclosures to improve modeling of individual characteristics and prediction of annotator labels for subjective tasks. The volume of personal information has historically been restricted and thus little exploration has gone into understanding what kind of information is most informative for predicting annotator labels. In this work, we categorize self-disclosure sentences and use them to build annotator models for predicting judgments of social norms. We perform several ablations and analyses to examine the impact of the type of information on our ability to predict annotation patterns. We find that demographics are more impactful than attitudes, relationships, and experiences. Generally, theory-based approaches worked better than automatic clusters. Contrary to previous work, only a small number of related comments are needed. Lastly, having a more diverse sample of annotator self-disclosures leads to the best performance.

Figures (15)

• Figure 1: Illustration of the experimental setup testing how types of self-disclosure information about annotators (e.g., demographics, experiences, attitudes) influence a language model’s ability to predict their judgments in social dilemmas. The model uses Reddit posts as training data and an embedding of commenter self-disclosures to make a decision.
• Figure 2: Percentage of times that each similar sentence falls into each theory based category (top, ) and automatic cluster (bottom, ) for experiments in Table \ref{['tab:samples——sit']}. Most similar annotator comments do not belong to a category, as 75% of similar posts have no theory based category and 78% have no automatic category.
• Figure 3: Distribution plots showing the percent coverage of author comments (left) in the top 5 most similar comments selected from all comments (Table \ref{['tab:samples——sit']}) and the ratio between first and second rank counts of most frequently selected similar comments (right). Medians are printed below the median line.
• Figure 4: Proportion of times in the training data that one of the top 5 most similar posts was selected from each subcategory, separately for demographics (left, ) and experiences (right, ), for top two rows of Table \ref{['tab:situtation_results']}.
• Figure 5: Results of kMeans clustering algorithm