Extracting triples from dialogues for conversational social agents

TL;DR

This work addresses the sparse explicit representation of social knowledge in dialogue by extracting RDF-style triples from conversations to support episodic memory and transparent, controllable agents. It introduces turn- and conversation-level datasets, and evaluates five extraction approaches spanning rule-based CFG, dependency-pattern methods, OpenIE, fine-tuned BERT models, and few-shot prompting with Llama, highlighting the unique challenges of co-reference, ellipsis, and discourse phenomena in dialogue. The results show that structure-based methods perform best on simple turns while neural methods better handle complex predicates, though conversation-level extraction remains significantly harder than turn-level extraction. The study provides publicly available data and models to spur future work in explicit social knowledge representation and reasoning in conversational AI.

Abstract

Obtaining an explicit understanding of communication within a Hybrid Intelligence collaboration is essential to create controllable and transparent agents. In this paper, we describe a number of Natural Language Understanding models that extract explicit symbolic triples from social conversation. Triple extraction has mostly been developed and tested for Knowledge Base Completion using Wikipedia text and data for training and testing. However, social conversation is very different as a genre in which interlocutors exchange information in sequences of utterances that involve statements, questions, and answers. Phenomena such as co-reference, ellipsis, coordination, and implicit and explicit negation or confirmation are more prominent in conversation than in Wikipedia text. We therefore describe an attempt to fill this gap by releasing data sets for training and testing triple extraction from social conversation. We also created five triple extraction models and tested them in our evaluation data. The highest precision is 51.14 for complete triples and 69.32 for triple elements when tested on single utterances. However, scores for conversational triples that span multiple turns are much lower, showing that extracting knowledge from true conversational data is much more challenging.