Probing the contents of semantic representations from text, behavior, and brain data using the psychNorms metabase

TL;DR

The paper investigates whether semantic representations from text, behavior, and brain data encode overlapping or distinct information, using representational similarity analysis (RSA) and a novel representational content analysis (RCA). It introduces the psychNorms metabase of 292 human-rated norms to interpret representation content and demonstrates that behavior-based representations can rival or exceed text in capturing psychological dimensions, while also contributing unique variance in affective, agentic, and socio-moral domains. A controlled vocabulary approach with a base set $V_ ext{base}$ enables cross-type RSA, and ensemble RCA shows combining text and behavior often improves norm explainability, suggesting that behavior complements text for human-aligned semantic representations. The findings have implications for evaluating and aligning large language models and for enriching cognitive and affective modeling with behavior-derived semantic information.

Abstract

Semantic representations are integral to natural language processing, psycholinguistics, and artificial intelligence. Although often derived from internet text, recent years have seen a rise in the popularity of behavior-based (e.g., free associations) and brain-based (e.g., fMRI) representations, which promise improvements in our ability to measure and model human representations. We carry out the first systematic evaluation of the similarities and differences between semantic representations derived from text, behavior, and brain data. Using representational similarity analysis, we show that word vectors derived from behavior and brain data encode information that differs from their text-derived cousins. Furthermore, drawing on our psychNorms metabase, alongside an interpretability method that we call representational content analysis, we find that, in particular, behavior representations capture unique variance on certain affective, agentic, and socio-moral dimensions. We thus establish behavior as an important complement to text for capturing human representations and behavior. These results are broadly relevant to research aimed at learning human-aligned semantic representations, including work on evaluating and aligning large language models.

Figures (5)

• Figure 1: An illustration of our approach. Word vectors are first obtained from the different data sources and then compared via representational similarity analysis (RSA) and representational content analysis (RCA).
• Figure 2: An illustration of the size of the vocabularies (y-axis, log-scaled) for each representation and norm (x-axis, grouped into higher-level categories) used in our analyses. The representations have been grouped into each data type (text, behavior, and brain).
• Figure 3: A: A 2-dimensional projection of the representational similarity space. The space was obtained by multidimensional scaling of the pairwise Spearman dissimilarity matrix between representations. Text = green, behavior = purple, brain = blue. B: A heatmap visualization of the pairwise Spearman similarity matrix.
• Figure 4: Average 5-fold cross-validation (pseudo-)$R^2$ test performance for text, behavior, and brain representations (rows, grouped) on 292 norms grouped into 27 norm categories (columns). Performances are aggregated by first taking the mean $R^2$ on each norm and then the median of the norm-wise (mean) $R^2$s for each norm category. Representations are ordered within each data type in terms of overall performance. Norm categories are ordered in terms of the performance of the top-performing behavior representation (PPMI SVD SWOW). Missing values are the result of an insufficient number of test samples.
• Figure 5: Average 5-fold cross-validation (pseudo-)$R^2$ performance for the top-2 Text (CBOW GoogleNews and fastText CommonCrawl), top Behavior (PPMI SVD SWOW) representations, and all Text & Text and Text & Behavior ensemble combinations. Performances are aggregated by first taking the mean (difference in) $R^2$ on each norm and then the median of the norm-wise (mean) $R^2$ for each norm category. Norms are ordered in terms of the performance of Text & Behavior. Emboldened differences (Text & Behavior - Text & Text) reflect a Wilcoxon signed rank $p<.05$.