Leveraging Machine-Generated Rationales to Facilitate Social Meaning Detection in Conversations

TL;DR

This paper introduces a generalizable framework that leverages LLM-generated rationales to uncover implicit social meaning in conversations and improve detection tasks such as emotion recognition (ERC) and resistance strategies (RES). It designs a multi-faceted prompting approach to elicit three rationale types—intention, assumptions, and implicit information—which are then used to augment dialogue text during training and inference. Across two tasks and four datasets, the approach yields significant improvements in both in-domain and cross-domain transfer, with larger gains in low-data regimes and for ERC. The work highlights the potential of rationale-augmented dialogue understanding and provides public resources to foster further research, while noting limitations such as dependence on proprietary LLMs and prompt sensitivity.

Abstract

We present a generalizable classification approach that leverages Large Language Models (LLMs) to facilitate the detection of implicitly encoded social meaning in conversations. We design a multi-faceted prompt to extract a textual explanation of the reasoning that connects visible cues to underlying social meanings. These extracted explanations or rationales serve as augmentations to the conversational text to facilitate dialogue understanding and transfer. Our empirical results over 2,340 experimental settings demonstrate the significant positive impact of adding these rationales. Our findings hold true for in-domain classification, zero-shot, and few-shot domain transfer for two different social meaning detection tasks, each spanning two different corpora.

Figures (10)

• Figure 1: Fraction of cases where the classification performance was significantly better, same, or worse, when rationales were augmented, for two different tasks, i.e. detecting resisting strategies (RES) and recognizing emotions (ERC) and for two settings i.e., in-domain (ID) and transfer (TF).
• Figure 2: We present the prompting framework employed in this work to generate rationales that are subsequently used for dialogue understanding and transfer using pre-existing LLMs such as GPT-3.5-turbo and LLama-2 variants. We feed in the prompt (green box on the left) for a given dialogue to generate the speaker's intentions (INT), assumptions (ASM), and the underlying implicit information (IMP) (gray box in the right). For lack of space we showcase the generated rationales only for the first (in blue) and last utterance(in red).
• Figure 3: We present here the label distribution for the emotion recognition and the resisting strategies datasets.
• Figure 4: Here we illustrate the process of transfer from the source to target. The model is first fine-tuned on the source dialogues, which comprises the current utterance, the previous dialogue context, and the rationales (INT, ASM, and IMP for intentions, assumptions, and implicit information respectively). This fine-tuned model can then be used off-the-shelf for predictions on the target (zero-shot) or further fine-tuned in a few-shot setting.
• Figure 5: Performance of the base-variants of models (BERT, GPT2, and T5) on the four datasets for different few-shot examples. The solid and dashed lines correspond to the indomain (ID) and transfer (TF) case respectively.