Log2Motion: Biomechanical Motion Synthesis from Touch Logs

TL;DR

Log2Motion addresses the challenge of inferring realistic human motion from touch logs by tightly integrating a software emulator with a musculoskeletal forward simulator to produce biomechanically plausible finger–arm trajectories. The approach casts motion synthesis as learning muscle activations via reinforcement learning with a structured reward and decomposed motor operators (tapping and swiping), anchored by a Screen Mirror that links real applications to the physics environment. Empirical evaluations show the synthesized motions exhibit human-like kinematics, adhere to established speed–accuracy trade-offs (e.g., Fitts’ Law), align with motion-capture data, and yield meaningful estimates of motion, speed, accuracy, and effort. The work enables large-scale ergonomic and usability assessments from log data, supports adaptive postures, and offers a platform for downstream applications in accessibility and design, all while maintaining open science practices with code available at the project repository. $R_T$ values and other reward components guide task completion, balancing intermediate objectives with a dominant terminal goal to ensure robust learning.

Abstract

Touch data from mobile devices are collected at scale but reveal little about the interactions that produce them. While biomechanical simulations can illuminate motor control processes, they have not yet been developed for touch interactions. To close this gap, we propose a novel computational problem: synthesizing plausible motion directly from logs. Our key insight is a reinforcement learning-driven musculoskeletal forward simulation that generates biomechanically plausible motion sequences consistent with events recorded in touch logs. We achieve this by integrating a software emulator into a physics simulator, allowing biomechanical models to manipulate real applications in real-time. Log2Motion produces rich syntheses of user movements from touch logs, including estimates of motion, speed, accuracy, and effort. We assess the plausibility of generated movements by comparing against human data from a motion capture study and prior findings, and demonstrate Log2Motion in a large-scale dataset. Biomechanical motion synthesis provides a new way to understand log data, illuminating the ergonomics and motor control underlying touch interactions.

Figures (12)

• Figure 1: In motion synthesis, a biomechanical body model generates motions that are aligned with sequences of events recorded in a log file. These log files can stem from user interactions or expert demonstrations. Log2Motion introduces three key innovations: (1) a problem formulation that defines motion synthesis as a POMDP, (2) an integrated physics simulator that captures both biomechanical and display dynamics, and (3) a reward model that supports learning plausible movement policies for dexterous touch interactions using .
• Figure 2: Simulation results showing Fitts’ law speed–accuracy trade-off for the three tapping policies accurate, normal, and fast. Each point corresponds to a button size diameter (14--4 mm). The results show strong adherence to the linear relationship of movement time and task difficulty as described by Fitts' law.
• Figure 3: Movement times for the fast tapping operator in a 1D task at distances of 20 mm and 30 mm with different button widths (2.4 mm, 4.8 mm, 7.2 mm). Simulation results are presented as mean movement times with 95% confidence intervals based on 200 repeated runs. For the human data, only mean values were available upon request. The simulated mean movement times align closely with the human data across all conditions, consistent with results reported in bi_ffitts_2013.
• Figure 4: Heatmaps of fingertip positions during interaction with the touchscreen (200 simulation runs). For the tapping motor operators accurate, normal, and fast, the heatmaps show activations within a 10 mm diameter button, while the swipe policy represents finger movement toward the target. The heatmaps clearly show a bigger spread of endpoint contacts for the fast operator compared to the normal and accurate operators.
• Figure 5: Velocity profiles of the accurate, normal, and fast tapping operator and the swiping operator during interaction with a target of diameter 10 mm. Left: overall velocity profile. Right: velocity profile measured on the display surface for Normal Swipe policy, ending with the lift off the finger. The velocity profiles show strong similarities to the ballistic movements observed in human pointing behavior.