Towards Senior-Robot Interaction: Reactive Robot Dog Gestures

TL;DR

Facing loneliness in aging populations, this work develops a senior-oriented quadruped robot framework that combines MediaPipe-based gesture recognition with curriculum-based reinforcement learning to generate socially expressive gestures. It demonstrates a complete pipeline from gesture input to RL-trained outputs in simulation (Isaac Gym) and real hardware (Unitree Go1), revealing strong sim-to-real performance yet highlighting hardware-induced discrepancies. The contributions include an end-to-end gesture control system, a repertoire of social gestures (e.g., paw lift, pointing, waving), and a structured plan for sim-to-real adaptation and future user studies. This work advances senior-robot interaction by enabling intuitive input and legible, socially meaningful robot expressions tailored for elderly users.

Abstract

As the global population ages, many seniors face the problem of loneliness. Companion robots offer a potential solution. However, current companion robots often lack advanced functionality, while task-oriented robots are not designed for social interaction, limiting their suitability and acceptance by seniors. Our work introduces a senior-oriented system for quadruped robots that allows for more intuitive user input and provides more socially expressive output. For user input, we implemented a MediaPipe-based module for hand gesture and head movement recognition, enabling control without a remote. For output, we designed and trained robotic dog gestures using curriculum-based reinforcement learning in Isaac Gym, progressing from simple standing to three-legged balancing and leg extensions, and more. The final tests achieved over 95\% success on average in simulation, and we validated a key social gesture (the paw-lift) on a Unitree robot. Real-world tests demonstrated the feasibility and social expressiveness of this framework, while also revealing sim-to-real challenges in joint compliance, load distribution, and balance control. These contributions advance the development of practical quadruped robots as social companions for the senior and outline pathways for sim-to-real adaptation and inform future user studies.

Figures (6)

• Figure 1: System architecture: A MediaPipe-based module processes user gestures into commands, which are sent via TCP to a server. The robot runs a high-level controller for posture and simple locomotion via API calls or a low-level controller that executes complex social gestures using curriculum-learning reinforcement-learning (CL-RL) policies.
• Figure 2: Examples of gesture detection under dim lighting. (a) Correct recognition of an open palm. (b) Misclassification of pointing up as a fist.
• Figure 3: Training performance across random seeds for Stage 2 (leg lift). Top: reward evolution (mean $\pm$ 1 s.d., $N{=}12$). Bottom: final success rates per seed (range 94.5%–99.7%, mean $98.6\%\pm1.3\%$). Note: the x-axis shows normalized training progress [0–1], while reward values are unnormalized (raw scale).
• Figure 4: Overview of three gesture primitives executed by the quadruped robot.
• Figure 5: Time-series data from the ablation study. (a) Direct training results in unstable behaviour, with erratic contact forces and orientation, leading to an early fall. (b) Our curriculum-based method achieves stable height tracking, consistent tripod contact forces (FL near zero), and a learned, steady roll angle for balance. For clarity of comparison, all reward curves are normalised to the range [0, 1].