Social Orientation: A New Feature for Dialogue Analysis

TL;DR

It is shown that social orientation tags improve task performance, especially in low-resource settings, on both English and Chinese language benchmarks, and how social orientation tags help explain the outcomes of social interactions when used in neural models.

Abstract

There are many settings where it is useful to predict and explain the success or failure of a dialogue. Circumplex theory from psychology models the social orientations (e.g., Warm-Agreeable, Arrogant-Calculating) of conversation participants and can be used to predict and explain the outcome of social interactions. Our work is novel in its systematic application of social orientation tags to modeling conversation outcomes. In this paper, we introduce a new data set of dialogue utterances machine-labeled with social orientation tags. We show that social orientation tags improve task performance, especially in low-resource settings, on both English and Chinese language benchmarks. We also demonstrate how social orientation tags help explain the outcomes of social interactions when used in neural models. Based on these results showing the utility of social orientation tags for dialogue outcome prediction tasks, we release our data sets, code, and models that are fine-tuned to predict social orientation tags on dialogue utterances.

Figures (8)

• Figure 1: Distribution of GPT-4 labeled social orientation tags in the Conversations Gone Awry (CGA) Corpus.
• Figure 2: Confusion matrix comparing predicted social orientation tags to GPT-4 labels on the CGA test set. We note that many of the misclassifications occur among neighboring tags.
• Figure 3: Data ablation results showing average accuracy scores ($\pm$ 1 standard deviation) over 5 runs for various methods on the CGA (top) and CGA CMV (bottom) data sets. We see that social orientation features (Logistic (Social Counts) - Predicted) outperform text-only methods in low-resource regimes and that social orientation features help model accuracy, even in high resource settings.
• Figure 4: Data ablation results showing average accuracy scores ($\pm$ 1 standard deviation) over 5 runs for various methods on the constructed Chinese data set. We see high accuracy logistic regression results in low-resource settings and that text plus social orientation features outperform all other methods in high-resource settings.
• Figure 5: Complete set of interventions for explainability experiments.