Inducing Human-like Biases in Moral Reasoning Language Models

TL;DR

While larger models generally performed better on both metrics, BrainScores did not significantly improve after fine-tuning, and the accuracy on the ETHICS benchmark and the BrainScores between model activations and fMRI data did not significantly improve.

Abstract

In this work, we study the alignment (BrainScore) of large language models (LLMs) fine-tuned for moral reasoning on behavioral data and/or brain data of humans performing the same task. We also explore if fine-tuning several LLMs on the fMRI data of humans performing moral reasoning can improve the BrainScore. We fine-tune several LLMs (BERT, RoBERTa, DeBERTa) on moral reasoning behavioral data from the ETHICS benchmark [Hendrycks et al., 2020], on the moral reasoning fMRI data from Koster-Hale et al. [2013], or on both. We study both the accuracy on the ETHICS benchmark and the BrainScores between model activations and fMRI data. While larger models generally performed better on both metrics, BrainScores did not significantly improve after fine-tuning.

Figures (20)

• Figure 1: Accuracy values for the Commonsense split of the ETHICS dataset hendrycks_aligning_2021. See Table \ref{['tab:finetuned_commonsense']} for a tabular depiction of the data.
• Figure 2: Graphical depiction of the different sampling methods' effect on accuracy on the Commonsense split of ETHICS hendrycks_aligning_2021.
• Figure 3: Brain scores across the hidden layers from bert-large-cased, roberta-large, and deberta-v2-xlarge across our different fine-tuning protocols.
• Figure 4: Subject and layer averaged CoD taken from bert-large-cased A) without fine-tuning on ETHICS or the fMRI recordings, B) with fine-tuning on ETHICS and fMRI recordings, and C) their difference.
• Figure 5: Subject and layer averaged CoD taken from roberta-large A) without fine-tuning on ETHICS or the fMRI recordings, B) with fine-tuning on ETHICS but not the fMRI recordings, and C) their difference.