Designing for Human-Agent Alignment: Understanding what humans want from their agents

TL;DR

This work addresses the question of what aspects humans want aligned in autonomous agents that negotiate on their behalf. It employs a qualitative think-aloud study with 10 participants using prompt-engineered seller/buyer agents to simulate online marketplace negotiations, analyzed via open coding and abductive thematic analysis. The authors identify six core dimensions of Human-Agent Alignment (Knowledge Schema, Autonomy/Agency, Operational Training, Reputational Heuristics, Ethics, and Human Engagement) and propose three design directions to guide designers in building alignment-aware agents. The findings underscore the need for longitudinal, user-customizable alignment and highlight reputational and safety risks that arise when alignment is incomplete, with practical implications for UI/UX and governance of human-agent collaborations.

Abstract

Our ability to build autonomous agents that leverage Generative AI continues to increase by the day. As builders and users of such agents it is unclear what parameters we need to align on before the agents start performing tasks on our behalf. To discover these parameters, we ran a qualitative empirical research study about designing agents that can negotiate during a fictional yet relatable task of selling a camera online. We found that for an agent to perform the task successfully, humans/users and agents need to align over 6 dimensions: 1) Knowledge Schema Alignment 2) Autonomy and Agency Alignment 3) Operational Alignment and Training 4) Reputational Heuristics Alignment 5) Ethics Alignment and 6) Human Engagement Alignment. These empirical findings expand previous work related to process and specification alignment and the need for values and safety in Human-AI interactions. Subsequently we discuss three design directions for designers who are imagining a world filled with Human-Agent collaborations.