User Agency and System Automation in Interactive Intelligent Systems

TL;DR

This thesis addresses the challenge of balancing user agency with system automation in interactive intelligent systems that share control over a variable. It advances this goal through four core contributions: (1) Omni-v1 and (2) Omni-v2, spherical electromagnets enabling untethered yet grounded haptic feedback with integrated sensing to reduce external tracking, (3) MagPen, an MPCC-based optimal-control framework that gently guides a pen while preserving user autonomy, and (4) MARLUI, a model-free multi-agent RL approach for adaptive user interfaces that learns interface policies without relying on heuristics or real user data. The work demonstrates that shared control can outperform naive strategies and shows how explicit or implicit human behavior models can be embedded into control designs to better account for user agency. By combining innovative hardware design, fast magnetic-field modeling, and both model-based and model-free control strategies, it lays groundwork for end-to-end intelligent interaction systems that are intuitive, efficient, and adaptable across tasks such as drawing, sculpting, gaming, and UI adaptation. The findings highlight practical implications for AR/VR, design tools, and interactive interfaces, revealing a pathway toward integrated algorithmic, engineering, and design perspectives in human-AI collaboration, while outlining future directions in human sensing, theory-of-mind user models, and hybrid control strategies.

Abstract

Balancing user agency and system automation is essential for effective human-AI interactions. Fully automated systems can deliver efficiency but risk undermining usability and user autonomy, while purely manual tools are often inefficient and fail to enhance user capabilities. This dissertation addresses the question: "How can we balance user agency and system automation for interactions with intelligent systems?" We present four main contributions. First, we develop a spherical electromagnet that provides adjustable forces on an untethered tool, allowing haptic feedback while preserving user agency. Second, we create an integrated sensing and actuation system that tracks a passive magnetic tool in 3D and delivers haptic feedback without external tracking. Third, we propose an optimal control method for electromagnetic haptic guidance that balances user input with system control, enabling users to adjust trajectories and speed. Finally, we introduce a model-free reinforcement learning approach for adaptive interfaces that learns interface adaptations without heuristics or real user data. Our simulations and user studies show that shared control significantly outperforms naive strategies. By incorporating explicit or implicit models of human behavior into control strategies, intelligent systems can better account for user agency. We demonstrate that the trade-off between agency and automation is both an algorithmic challenge and an engineering concern, shaped by the design of physical devices and user interfaces. We advocate an integrated, end-to-end approach-combining algorithmic, engineering, and design perspectives-to enable more intuitive and effective interactions with intelligent systems.

Figures (46)

• Figure 1: Human-Computer Interaction as a closed-loop system, adapted from murray2018control
• Figure 2: The Haptic-Loop allows for bi-directional exchange of information between machine and user (hayward2004haptic).
• Figure 3: Diagram of the coordinate system used in the models
• Figure 4: An overview of the control problem. An agent/system takes action $\mathbf{a}$ or input $\mathbf{u}$ based on the state $\mathbf{x}$ or $\mathbf{s}$. The action/input updates the state. Furthermore the agent receives a reward $r$.
• Figure 5: We introduce a novel contact-free mechanism to render haptic feedback onto a tracked stylus via a hemispherical electromagnet. An approximate model of the magnet interaction and a computationally efficient control strategy allow for the dynamic rendering of attracting and repulsive forces, for example, allowing users to explore virtual surfaces in a thin shell surrounding the device (inset).