Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Moral Foundations Reddit Corpus

Jackson Trager, Alireza S. Ziabari, Elnaz Rahmati, Aida Mostafazadeh Davani, Preni Golazizian, Farzan Karimi-Malekabadi, Ali Omrani, Zhihe Li, Brendan Kennedy, Nils Karl Reimer, Melissa Reyes, Kelsey Cheng, Mellow Wei, Christina Merrifield, Arta Khosravi, Evans Alvarez, Morteza Dehghani

TL;DR

MFRC extends the Moral Foundations Twitter Corpus by providing a Reddit-based dataset (16,123 comments from 12 subreddits) annotated for 8 moral sentiment categories under the revised Moral Foundations Theory, including annotator metadata to enable bias analysis. The authors benchmark a range of baselines, from zero-shot and few-shot prompting of LLMs (Llama3-8B, Ministral-8B) to parameter-efficient and full fine-tuning of encoder models like BERT, using weighted losses for label imbalance. They find that fine-tuned encoders outperform LLM baselines on this subjective task, with cross-corpus experiments showing partial transfer to the MFTC dataset, underscoring the value of human-annotated moral corpora for AI alignment. The MFRC thus serves as a cross-platform benchmark for multi-label moral sentiment classification and enables richer cross-domain analyses of online moral rhetoric.

Abstract

Moral framing and sentiment can affect a variety of online and offline behaviors, including donation, environmental action, political engagement, and protest. Various computational methods in Natural Language Processing (NLP) have been used to detect moral sentiment from textual data, but achieving strong performance in such subjective tasks requires large, hand-annotated datasets. Previous corpora annotated for moral sentiment have proven valuable, and have generated new insights both within NLP and across the social sciences, but have been limited to Twitter. To facilitate improving our understanding of the role of moral rhetoric, we present the Moral Foundations Reddit Corpus, a collection of 16,123 English Reddit comments that have been curated from 12 distinct subreddits, hand-annotated by at least three trained annotators for 8 categories of moral sentiment (i.e., Care, Proportionality, Equality, Purity, Authority, Loyalty, Thin Morality, Implicit/Explicit Morality) based on the updated Moral Foundations Theory (MFT) framework. We evaluate baselines using large language models (Llama3-8B, Ministral-8B) in zero-shot, few-shot, and PEFT settings, comparing their performance to fine-tuned encoder-only models like BERT. The results show that LLMs continue to lag behind fine-tuned encoders on this subjective task, underscoring the ongoing need for human-annotated moral corpora for AI alignment evaluation. Keywords: moral sentiment annotation, moral values, moral foundations theory, multi-label text classification, large language models, benchmark dataset, evaluation and alignment resource

The Moral Foundations Reddit Corpus

TL;DR

MFRC extends the Moral Foundations Twitter Corpus by providing a Reddit-based dataset (16,123 comments from 12 subreddits) annotated for 8 moral sentiment categories under the revised Moral Foundations Theory, including annotator metadata to enable bias analysis. The authors benchmark a range of baselines, from zero-shot and few-shot prompting of LLMs (Llama3-8B, Ministral-8B) to parameter-efficient and full fine-tuning of encoder models like BERT, using weighted losses for label imbalance. They find that fine-tuned encoders outperform LLM baselines on this subjective task, with cross-corpus experiments showing partial transfer to the MFTC dataset, underscoring the value of human-annotated moral corpora for AI alignment. The MFRC thus serves as a cross-platform benchmark for multi-label moral sentiment classification and enables richer cross-domain analyses of online moral rhetoric.

Abstract

Moral framing and sentiment can affect a variety of online and offline behaviors, including donation, environmental action, political engagement, and protest. Various computational methods in Natural Language Processing (NLP) have been used to detect moral sentiment from textual data, but achieving strong performance in such subjective tasks requires large, hand-annotated datasets. Previous corpora annotated for moral sentiment have proven valuable, and have generated new insights both within NLP and across the social sciences, but have been limited to Twitter. To facilitate improving our understanding of the role of moral rhetoric, we present the Moral Foundations Reddit Corpus, a collection of 16,123 English Reddit comments that have been curated from 12 distinct subreddits, hand-annotated by at least three trained annotators for 8 categories of moral sentiment (i.e., Care, Proportionality, Equality, Purity, Authority, Loyalty, Thin Morality, Implicit/Explicit Morality) based on the updated Moral Foundations Theory (MFT) framework. We evaluate baselines using large language models (Llama3-8B, Ministral-8B) in zero-shot, few-shot, and PEFT settings, comparing their performance to fine-tuned encoder-only models like BERT. The results show that LLMs continue to lag behind fine-tuned encoders on this subjective task, underscoring the ongoing need for human-annotated moral corpora for AI alignment evaluation. Keywords: moral sentiment annotation, moral values, moral foundations theory, multi-label text classification, large language models, benchmark dataset, evaluation and alignment resource
Paper Structure (17 sections, 1 equation, 1 figure, 7 tables)

This paper contains 17 sections, 1 equation, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Corpus Overview
  3. General Sampling Procedure
  4. Annotation
  5. Moral sentiment
  6. Annotators
  7. Annotator Metadata
  8. Annotation Results
  9. Baseline Classification Models
  10. Zero-shot and Few-shot Prompting
  11. Classification Fine-tuning
  12. Results
  13. Cross-Corpus Classification
  14. Discussion
  15. Data Disclaimer
  16. ...and 2 more sections

Figures (1)

  • Figure 1: The heatmaps show Interannotator Agreement (PABAK and Kappa) scores for all subreddits and foundations. Higher agreement corresponds with darker colors in both heatmaps.