Proactive User Information Acquisition via Chats on User-Favored Topics

TL;DR

The paper introduces PIVOT, a task to proactively acquire user information through chats on user-favored topics, highlighting the challenge that modern LLMs struggle to do this smoothly: information can be gleaned in most chats, but systems often produce abrupt utterances, and non-abrupt success is low. To investigate, the authors build a 650-chat dataset and perform extensive analyses of abruptness, relationship types between TOPIC and QUESTION, and techniques like cushion utterances and explicit explanations. They propose a simple yet effective strategy-based, BDI-inspired system that uses a separate evaluator LLM to select non-abrupt responses and demonstrates a substantial rise in task success to about $40\%$, outperforming pure-prompt baselines. The work lays groundwork for more robust proactive dialogue systems, suggests directions for richer data-driven training, and discusses practical implications and limitations for real-world applications.

Abstract

Chat-oriented dialogue systems designed to provide tangible benefits, such as sharing the latest news or preventing frailty in senior citizens, often require Proactive acquisition of specific user Information via chats on user-faVOred Topics (PIVOT). This study proposes the PIVOT task, designed to advance the technical foundation for these systems. In this task, a system needs to acquire the answers of a user to predefined questions without making the user feel abrupt while engaging in a chat on a predefined topic. We found that even recent large language models (LLMs) show a low success rate in the PIVOT task. We constructed a dataset suitable for the analysis to develop more effective systems. Finally, we developed a simple but effective system for this task by incorporating insights obtained through the analysis of this dataset.