LIMIT: Learning Interfaces to Maximize Information Transfer

TL;DR

LIMIT tackles the problem of learning interpretable, personalized interfaces that convey hidden information from robots to humans by maximizing the conditional mutual information $I(a; \\theta \\mid s)$. It introduces a three-network online framework—$\\mathcal{H}_{\\phi}$, $\\mathcal{R}_{\\psi}$, and $\\Delta_{\\sigma}$—to maximize a tractable proxy of information gain through losses $\\mathcal{L}_{conv}$ and $\\mathcal{L}_{dist}$, while accounting for human co-adaptation with a bias toward recent data. The approach is modality-agnostic and demonstrated across simulated 1D/2D tasks, online user studies, and in-person experiments with auditory, visual, and haptic feedback, showing improved task performance and user-perceived interpretability compared to baselines like Naive and Bayes. By learning from interactions rather than relying on task models or pre-defined mappings, LIMIT enables personalized, co-adaptive interfaces that can transfer information more efficiently in diverse human-robot contexts, with potential applications in sensory substitution and BCIs. The work advances the design of intuitive, information-driven feedback mechanisms for complex human-in-the-loop systems, while outlining future work to fuse task-specific rewards with the information-transfer objective to ensure necessary, meaningful signaling.

Abstract

Robots can use auditory, visual, or haptic interfaces to convey information to human users. The way these interfaces select signals is typically pre-defined by the designer: for instance, a haptic wristband might vibrate when the robot is moving and squeeze when the robot stops. But different people interpret the same signals in different ways, so that what makes sense to one person might be confusing or unintuitive to another. In this paper we introduce a unified algorithmic formalism for learning co-adaptive interfaces from scratch. Our method does not need to know the human's task (i.e., what the human is using these signals for). Instead, our insight is that interpretable interfaces should select signals that maximize correlation between the human's actions and the information the interface is trying to convey. Applying this insight we develop LIMIT: Learning Interfaces to Maximize Information Transfer. LIMIT optimizes a tractable, real-time proxy of information gain in continuous spaces. The first time a person works with our system the signals may appear random; but over repeated interactions the interface learns a one-to-one mapping between displayed signals and human responses. Our resulting approach is both personalized to the current user and not tied to any specific interface modality. We compare LIMIT to state-of-the-art baselines across controlled simulations, an online survey, and an in-person user study with auditory, visual, and haptic interfaces. Overall, our results suggest that LIMIT learns interfaces that enable users to complete the task more quickly and efficiently, and users subjectively prefer LIMIT to the alternatives. See videos here: https://youtu.be/IvQ3TM1_2fA

Figures (13)

• Figure 1: Interface selecting signals to convey information to the human operator. Choosing the right feedback is challenging because the way people respond to signals varies across tasks, users, and interface types; e.g., when a person sees this LED pattern should they go left or right? We introduce a unified algorithmic framework that co-adapts to the current user by learning to pick signals that maximize information transfer.
• Figure 2: Simulation results in $1$D environment. (Left) Error is the distance between the human's final position and the phone location $\theta$. (Middle) Interfaces paired with the Rotate human. A repeated measures ANOVA reveals that the interface type had a significant effect on error ($F(4, 396) = 9.2$, $p < .001$), with LIMIT resulting less error than the alternatives ($p<.05$). (Right) Interfaces paired with the Align human. The interface algorithm affects error ($F(4, 396) = 16.8, p<.001$); pairwise comparisons show that LIMIT leads to less error than all alternatives besides Distinguish ($p<.05$).
• Figure 3: Simulation results in $2$D environment. (Left) The human observes vector $x$ and tries to reach hidden location $\theta$. (Middle) Results with Rotate human: differences here are not statistically significant. (Right) Interface paired with an Align human. Here interface type has a significant effect on error ($F(4,196)=9.0$, $p<.001$), and humans using LIMIT have less error by the final interaction than humans using Naive, Bayes, or Convey ($p<.001$).
• Figure 4: Simulation results in the $2$D environment when the signal $x$ and hidden information $\theta$ have different dimensions. (Left) The interface signal is $4$-dimensional, but only two dimensions are necessary to convey position $\theta$. Interface type has a significant effect on error ($F(4,196)=10.2$, $p<.001$) and LIMIT results in lower final error than either Naive or Convey ($p<.05$). (Right) Now the hidden information is $4$ dimensional, and the interface must embed this $\theta$ to a lower-dimensional signal $x$ (e.g., the interface is trying to convey two phone locations). Humans reach different errors with different methods ($F(4,196)=4.6$, $p<.001$), but LIMIT yields less error than all baselines ($p<.05$).
• Figure 5: (Left) Simulation results from the autonomous driving task. The autonomous car has four different driving policies, and must signal its current policy to the human in order to help the human driver avoid a collision. Interfaces generated by LIMIT result in a lower collision rate than either Naive or Bayes ($p < 0.001$). (Right) Results from a $10$-dimensional environment. This simulation extends Section \ref{['sec:sims-2d']} to a high-dimensional setting where the states, actions, signals, and hidden information are all $10$-dimensional vectors in a continuous space. Despite this increase in dimension, the interfaces generated by LIMIT still result in a lower error at the end of an interaction than those generated by Naive or Bayes ($p < 0.001$).