Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Language Models Understand Social Behavior in Clinical Conversations?

Manas Satish Bedmutha, Feng Chen, Andrea Hartzler, Trevor Cohen, Nadir Weibel

TL;DR

This work investigates whether foundation LLMs can infer a broad set of social signals from clinical transcripts without fine tuning. The authors introduce Social-LM, a pipeline that evaluates three diverse LLMs with four prompting strategies on 20 social signals drawn from the EF dataset, binarized for analysis. Across extensive analyses, LLaMA configurations generally outperform smaller models, with few-shot and chain-of-thought prompts offering notable gains, and an ensemble approach further improving results. They also examine signal difficulty, temporal dynamics within visits, and demographic disparities, identifying context and phase dependent effects and highlighting the need for uncertainty-aware, segment-aware SSP in healthcare settings.

Abstract

Effective communication between providers and their patients influences health and care outcomes. The effectiveness of such conversations has been linked not only to the exchange of clinical information, but also to a range of interpersonal behaviors; commonly referred to as social signals, which are often conveyed through non-verbal cues and shape the quality of the patient-provider relationship. Recent advances in large language models (LLMs) have demonstrated an increasing ability to infer emotional and social behaviors even when analyzing only textual information. As automation increases also in clinical settings, such as for transcription of patient-provider conversations, there is growing potential for LLMs to automatically analyze and extract social behaviors from these interactions. To explore the foundational capabilities of LLMs in tracking social signals in clinical dialogue, we designed task-specific prompts and evaluated model performance across multiple architectures and prompting styles using a highly imbalanced, annotated dataset spanning 20 distinct social signals such as provider dominance, patient warmth, etc. We present the first system capable of tracking all these 20 coded signals, and uncover patterns in LLM behavior. Further analysis of model configurations and clinical context provides insights for enhancing LLM performance on social signal processing tasks in healthcare settings.

Can Language Models Understand Social Behavior in Clinical Conversations?

TL;DR

This work investigates whether foundation LLMs can infer a broad set of social signals from clinical transcripts without fine tuning. The authors introduce Social-LM, a pipeline that evaluates three diverse LLMs with four prompting strategies on 20 social signals drawn from the EF dataset, binarized for analysis. Across extensive analyses, LLaMA configurations generally outperform smaller models, with few-shot and chain-of-thought prompts offering notable gains, and an ensemble approach further improving results. They also examine signal difficulty, temporal dynamics within visits, and demographic disparities, identifying context and phase dependent effects and highlighting the need for uncertainty-aware, segment-aware SSP in healthcare settings.

Abstract

Effective communication between providers and their patients influences health and care outcomes. The effectiveness of such conversations has been linked not only to the exchange of clinical information, but also to a range of interpersonal behaviors; commonly referred to as social signals, which are often conveyed through non-verbal cues and shape the quality of the patient-provider relationship. Recent advances in large language models (LLMs) have demonstrated an increasing ability to infer emotional and social behaviors even when analyzing only textual information. As automation increases also in clinical settings, such as for transcription of patient-provider conversations, there is growing potential for LLMs to automatically analyze and extract social behaviors from these interactions. To explore the foundational capabilities of LLMs in tracking social signals in clinical dialogue, we designed task-specific prompts and evaluated model performance across multiple architectures and prompting styles using a highly imbalanced, annotated dataset spanning 20 distinct social signals such as provider dominance, patient warmth, etc. We present the first system capable of tracking all these 20 coded signals, and uncover patterns in LLM behavior. Further analysis of model configurations and clinical context provides insights for enhancing LLM performance on social signal processing tasks in healthcare settings.
Paper Structure (23 sections, 2 equations, 3 figures, 6 tables)

This paper contains 23 sections, 2 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Social Signals in Clinical Conversations
  4. LLMs in patient-provider communication
  5. Modeling Social Behavior in Clinical Conversations: The Social-LM Framework
  6. Dataset and Task Formulation
  7. Prompt Design
  8. Model Selection and Experiments
  9. Evaluating LLM Performance on Social Signal Prediction
  10. RQ1: Can large language models capture social dynamics in doctor-patient conversations?
  11. RQ2: How do model configurations impact LLM performance?
  12. RQ3: Which signals are easy or difficult for LLMs to track?
  13. RQ4: How does the difficulty of LLM predictions vary across the visit timeline?
  14. RQ5: Does the LLMs’ understanding of conversations differ by patient demographics?
  15. RQ6: Are social behaviors expressed differently at various stages of a visit?
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Percentage of high social signal labels distribution across 3-minute segments across the entire sample
  • Figure 2: Histogram distribution of number of correct predictions per sample. We see that the correct predictions are normally distributed.
  • Figure 3: Demographic Parity Ratio between white (n=74) and non-white (n=17) patients. We see that most configurations follow the four-fifths rule (threshold = 0.8). However, we see that patient dominance, provider dominance, patient interactivity, provider irritation and patient irritation did show disparity between the two groups, mainly for LLaMA.