Learning "Partner-Aware" Collaborators in Multi-Party Collaboration

TL;DR

This work tackles the challenge of training LLM-based collaborators that can effectively partner with both humans and other agents in multi-party tasks. It introduces Interruptible Collaborative Roleplayer (ICR), a framework built on a Modified-Action MDP (MAMDP) and a counterfactual invariance objective to selectively incorporate helpful interventions while resisting misleading ones. Theoretical results bound the suboptimality gap in MAMDPs and justify the counterfactual regularization, while empirical evaluations on DeliData and the Weights Task show that ICR consistently outperforms baselines in both full-language and restricted (no-press) settings, achieving higher task success and greater convergence of group beliefs. The findings suggest that partner-aware learning with counterfactual grounding enhances robustness and common-ground formation in AI-assisted collaboration, with practical implications for educational and workplace tutoring contexts. Future work should explore more diverse intervention styles, human-in-the-loop data, and scalability to more complex multi-agent environments while addressing safety considerations around potential manipulation or deception.

Abstract

Large Language Models (LLMs) are increasingly bring deployed in agentic settings where they act as collaborators with humans. Therefore, it is increasingly important to be able to evaluate their abilities to collaborate effectively in multi-turn, multi-party tasks. In this paper, we build on the AI alignment and safe interruptability literature to offer novel theoretical insights on collaborative behavior between LLM-driven collaborator agents and an intervention agent. Our goal is to learn an ideal partner-aware collaborator that increases the group's common-ground (CG)-alignment on task-relevant propositions-by intelligently collecting information provided in interventions by a partner agent.We show how LLM agents trained using standard RLHF and related approaches are naturally inclined to ignore possibly well-meaning interventions, which makes increasing group common ground non-trivial in this setting. We employ a two-player Modified-Action MDP to examine this suboptimal behavior of standard AI agents, and propose Interruptible Collaborative Roleplayer (ICR)-a novel partner-aware learning algorithm to train CG-optimal collaborators. Experiments on multiple collaborative task environments show that ICR, on average, is more capable of promoting successful CG convergence and exploring more diverse solutions in such tasks.

Figures (11)

• Figure 1: (a) Cumulative CG (common-ground) score of baselines for equality (left), inequality (middle), and order (right) propositions over block weights averaged across 100 dialogue trials across 15 turns in the “full-press” Weights Task. ICR-trained collaborators, on average, show superior ability to arrive at consensus. (b) Ablation test on the Delidata tracking batch-wise proxy reward during training of ICR collaborator over 8k steps with varying $\lambda_\text{Intent}$ values across 3 random seeds.
• Figure 2: We use GPT-4o as the expert collaborator to generate one turn of dialogue in the Wason Card Selection task, based on prior interaction over 14 turns of the game. \ref{['fig:final-turn_deli']} shows the 15th turn where the collaborator must provide a final solution for the group in the task. Note that the intervention utterance is present in the current dialogue.
• Figure 3: We use GPT-4o as an expert intervention agent to enhance collaborative reasoning in the Weights Task khebour-etal-2024-common. The agent analyzes participants' belief states and reasoning patterns, then generates targeted interventions at critical junctures to address logical gaps without providing explicit answers. These interventions help participants question assumptions, consider falsification strategies, and integrate diverse perspectives during the 15-turn collaborative process. Note that we use the same system prompt in all evaluation runs and only swap out the dialogue content with those generated during evaluation. We use $T=0$ and top-$p$ of 0.9 for sampling from GPT-4o.
• Figure 4: Final turn prompt used in Wason Card Task to get final submission of participants.
• Figure 5: We use GPT-4o as an expert intervention agent to improve collaborative reasoning on the Wason Card Selection task karadzhov2023delidata. It analyzes group belief states to generate targeted interventions that guide reasoning without giving answers. Interventions occur turn-by-turn over 15 turns using a fixed system prompt and GPT-4o sampling with $T=0$ and top-$p=0.9$.

Theorems & Definitions (11)

• Example 1: DeliData Wason Card Task
• Lemma 3.1: Bellman Optimality of Preference-Aligned Collaborators
• Theorem 3.2: Suboptimality of Preference-Aligned Collaborators
• Lemma B.1: Bellman Optimality of Preference-Aligned Collaborators (Detailed)
• proof
• Lemma B.2: Token-to-Intervention Bellman Optimality for Collaborator Agents
• proof
• Theorem B.3: Suboptimality of Preference-Aligned Collaborators
• proof
• Theorem B.4: Counterfactual Invariance Bounds Suboptimality