Learning User Preferences Through Interaction for Long-Term Collaboration

TL;DR

The paper tackles learning and leveraging user preferences across multiple collaborative sessions. It introduces MultiSessionCollab, a benchmark where agents must persist and refine user preferences through memory and session-level reflections, guided by a reinforcement learning framework (GRPO) that rewards comprehensive, non-hallucinatory reflections. Empirical results show memory-equipped agents achieve higher task success, shorter conversations, and reduced user effort, with gains amplified through RL-trained reflections; results are corroborated by a real-world user study demonstrating improved personalization and proactivity. The work provides a foundation for sustained human-AI collaboration, highlighting memory as a key driver for long-term alignment and proposing directions for more robust, cross-domain preference learning.

Abstract

As conversational agents accumulate experience collaborating with users, adapting to user preferences is essential for fostering long-term relationships and improving collaboration quality over time. We introduce MultiSessionCollab, a benchmark that evaluates how well agents can learn user preferences and leverage them to improve collaboration quality throughout multiple sessions. To develop agents that succeed in this setting, we present long-term collaborative agents equipped with a memory that persists and refines user preference as interaction experience accumulates. Moreover, we demonstrate that learning signals can be derived from user simulator behavior in MultiSessionCollab to train agents to generate more comprehensive reflections and update their memory more effectively. Extensive experiments show that equipping agents with memory improves long-term collaboration, yielding higher task success rates, more efficient interactions, and reduced user effort. Finally, we conduct a human user study that demonstrates that memory helps improve user experience in real-world settings.

Figures (10)

• Figure 1: The MultiSessionCollab benchmark with our long-term collaborative agent. Each session involves a user seeking help for a problem. The user maintains a draft answer that represents what they have learned from the interaction. They update their draft answer when the agent's responses are both helpful and preference-aligned. When responses violate preferences, the user enforces them, as indicated by the red text boxes. After each session, the agent reflects on the interaction to identify user preference information that will be useful for future interactions and update their memory accordingly. We measure collaboration quality in each session with task success, conversation length, and user effort.
• Figure 2: RL framework for improving session-level reflections. The policy model generates $n$ reflection rollouts for a conversation. The judge model evaluates each reflection against $\varepsilon$ (the subset of user utterances that enforce preferences) and assigns rewards. Advantages are computed and the policy is updated via GRPO.
• Figure 3: Performance across sessions for Llama-3.3-70B-Instruct. Each graph plots the delta between agents with memory and agents without memory across 20 sessions for Task Success ($\Delta_i^{TS}$) $\uparrow$, User Effort ($\Delta_i^{UE}$) $\downarrow$, and Conversation Length ($\Delta_i^{Len}$) $\downarrow$.
• Figure 4: Performance across sessions for Qwen-2.5-7B-Instruct (after GRPO). Each graph plots the delta between agents with memory and agents without memory across 20 sessions for Task Success ($\Delta_i^{TS}$), User Effort ($\Delta_i^{UE}$), and Conversation Length ($\Delta_i^{Len}$).
• Figure 5: Performance across sessions for Llama-3.1-8B-Instruct (after GRPO). Each graph plots the delta between agents with memory and agents without memory across 20 sessions for Task Success ($\Delta_i^{TS}$), User Effort ($\Delta_i^{UE}$), and Conversation Length ($\Delta_i^{Len}$).