The Muddy Waters of Modeling Empathy in Language: The Practical Impacts of Theoretical Constructs

TL;DR

This paper tackles the problem of inconsistent and underspecified empathy definitions in NLP by empirically analyzing how different theoretical groundings of empathy tasks transfer across tasks. Using an intermediate-task transfer framework with 18 empathy tasks and adapter-based RoBERTa models, it quantifies how construction granularity, measurement correspondence, and data conduciveness govern transfer performance. The results show that fine-grained, directly linked empathy definitions predict transfer better than abstract or adjacent constructs, while many transfers yield little or even negative gains, highlighting the risk of vague operationalizations. The authors argue for multidimensional, psychometrics-inspired empathy measures to support robust evaluation and resource development for empathy in language, potentially guiding dataset construction, annotation schemes, and downstream evaluation frameworks.

Abstract

Conceptual operationalizations of empathy in NLP are varied, with some having specific behaviors and properties, while others are more abstract. How these variations relate to one another and capture properties of empathy observable in text remains unclear. To provide insight into this, we analyze the transfer performance of empathy models adapted to empathy tasks with different theoretical groundings. We study (1) the dimensionality of empathy definitions, (2) the correspondence between the defined dimensions and measured/observed properties, and (3) the conduciveness of the data to represent them, finding they have a significant impact to performance compared to other transfer setting features. Characterizing the theoretical grounding of empathy tasks as direct, abstract, or adjacent further indicates that tasks that directly predict specified empathy components have higher transferability. Our work provides empirical evidence for the need for precise and multidimensional empathy operationalizations.

Figures (10)

• Figure 1: Experimental setup: How does intermediate task training on one empathy task impact performance on another target empathy task?
• Figure 2: Feature importances in regression fit to improvement over baseline calculated as the difference in $R^2$ when permuting the feature.
• Figure 3: Transfer performance by theme. Top: Significant improvement (left) and significant harm (right) counts for full data. Bottom: Insignificant difference or significant improvement (left) and significant harm (right) counts for limited data.
• Figure 4: Instructions for the empathy operationalization annotation task.
• Figure 5: Criteria provided to the annotators for scoring each aspect of the empathy construct along a 5-point Likert scale.