MyoInteract: A Framework for Fast Prototyping of Biomechanical HCI Tasks using Reinforcement Learning

TL;DR

MyoInteract tackles the barrier that biomechanical reinforcement learning poses for practical HCI prototyping by delivering GPU-accelerated, GUI-driven workflows that compress training from days to minutes. By decomposing tasks into composable primitives, exposing a multi-level, real-time diagnostic interface, and enforcing domain constraints, the framework enables rapid exploration of interaction designs while preserving biomechanical plausibility confirmed through Fitts'-law-type evaluations. A workshop with 12 researchers demonstrates that novices can configure and train biomechanical RL tasks within a single session, highlighting the framework’s potential to accelerate iteration cycles and broaden adoption beyond expert practitioners. Collectively, MyoInteract lowers barriers to entry and turns biomechanical RL into a practical, distributable tool for designing and evaluating physically plausible human-computer interactions across AR, public displays, and mobile contexts.

Abstract

Reinforcement learning (RL)-based biomechanical simulations have the potential to revolutionise HCI research and interaction design, but currently lack usability and interpretability. Using the Human Action Cycle as a design lens, we identify key limitations of biomechanical RL frameworks and develop MyoInteract, a novel framework for fast prototyping of biomechanical HCI tasks. MyoInteract allows designers to setup tasks, user models, and training parameters from an easy-to-use GUI within minutes. It trains and evaluates muscle-actuated simulated users within minutes, reducing training times by up to 98%. A workshop study with 12 interaction designers revealed that MyoInteract allowed novices in biomechanical RL to successfully setup, train, and assess goal-directed user movements within a single session. By transforming biomechanical RL from a days-long expert task into an accessible hour-long workflow, this work significantly lowers barriers to entry and accelerates iteration cycles in HCI biomechanics research.

Figures (12)

• Figure 1: The GUI of MyoInteract, a framework that allows HCI researchers to setup, train, monitor, and evaluate biomechanical simulations of interactive user behaviour. Users can (1) choose from a list of pre-defined task configurations for different HCI contexts (Mobile Touch, Public Display, Augmented Reality) and (2) adjust them, or define entirely new interaction tasks that involve sequences of target acquisition and selection. They can double-check the rendered interaction environment and (3) adjust the proposed reward function. With MyoInteract, training (4) takes less than one hour (previously: 12--48 hours) and (5) can be continuously monitored via success metrics. (6) Videos enable qualitative inspection of learned behaviour. Advanced mode (not shown) offers the option of setting additional parameters relating to task setup, the biomechanical model, and the RL training procedure.
• Figure 2: Workflow of the biomechanical RL simulations approach analysed using Norman's seven stages of action Norman2002-da, illustrating barriers in both the Gulf of Execution (task specification) and Gulf of Evaluation (training feedback). The bottom panels show how MyoInteract addresses each barrier: a GUI replaces manual XML/Python/YAML editing, GPU acceleration compresses training from 12–48 hours to under one hour, and a suite of performance metrics and visualisations replaces the minimal outputs of prior systems (which provided only aggregate metrics such as mean reward and episode length). These features significantly shorten the action cycle, transforming biomechanical RL simulation from a days-long expert endeavour into an accessible, hour-scale workflow.
• Figure 3: Box target settings in our GUI. Users can adjust position, size, minimum touch force, and orientation angle via sliders, and choose the object colour from a palette.
• Figure 4: Analysis of subtask completion metrics for each target in the AR task. Note how the success rates of subtasks build upon each other sequentially.
• Figure 5: Analysis of various reward components across training for the AR task. In this example, the individual reward components converge after 10-14 million steps, respectively.