"I Want It That Way": Enabling Interactive Decision Support Using Large Language Models and Constraint Programming

TL;DR

A novel approach, combining Large Language Models with Constraint Programming to facilitate interactive decision support and highlights the potential for a hybrid LLM and optimization approach for iterative preference elicitation, and suggests design considerations for building systems that support human-system collaborative decision-making processes.

Abstract

A critical factor in the success of decision support systems is the accurate modeling of user preferences. Psychology research has demonstrated that users often develop their preferences during the elicitation process, highlighting the pivotal role of system-user interaction in developing personalized systems. This paper introduces a novel approach, combining Large Language Models (LLMs) with Constraint Programming to facilitate interactive decision support. We study this hybrid framework through the lens of meeting scheduling, a time-consuming daily activity faced by a multitude of information workers. We conduct three studies to evaluate the novel framework, including a diary study (n=64) to characterize contextual scheduling preferences, a quantitative evaluation of the system's performance, and a user study (n=10) with a prototype system. Our work highlights the potential for a hybrid LLM and optimization approach for iterative preference elicitation and design considerations for building systems that support human-system collaborative decision-making processes.

Figures (9)

• Figure 1: An overview of the interactive loop for conversational decision support. The loop is initiated via an initial time suggestion (or set of suggestions) by the system presented to the user (A). During Preference Construction users evaluate a proposed suggestion (C) and either express a new preference (D) to improve the given suggestion, or accept the current suggestion. New preferences are integrated into the system during Preference Incorporation which requires the system to both embed the stated preference into the system (E) and then use it to generate a new time suggestion (F). This process is iterated until a suitable time is found.
• Figure 2: Breakdown of the percentage of diary study responses for (a) categories of contextual scheduling preferences and constraints, and (b) information missing from the calendar tool that inform the preference or constraint.
• Figure 3: Overview of MeetMate system architecture. Chat messages from users are translated into actions by the Constraint Manager Component (A), which selects from 5 actions including the Add Constraint Action (B) which translates natural language scheduling preferences into python functions. The system maintains an ordered list of scheduling constraints with priorities (C) that are used by the Constraint Programming Solver (D) to generate new time suggestions.
• Figure 4: Sample prompt for the constraint management component. Orange highlight reflects scheduling instance-specific inputs. Green highlights the output of the LLM.
• Figure 5: Three sample actions (represented by rows) taken by the constraint management component to manage constraint priorities. In each example, the left image shows the existing prioritized constraint list and current time suggestion. The middle image shows the new user message and the action taken by the constraint management component. The right image shows the resulting prioritized list of constraints and the new time suggestion. In the first example, a user specifies a new constraint with strong language (i.e., 'need') that is translated into a constraint with priority 1. The second example shows the same constraint with weaker language (i.e., 'if possible') that is translated into a constraint with priority 4. The final example shows a user specifiying that an unment constraint (Anton's attendance) needs to be met. The constraint manager changes the priority of that constraint to 1.