Multi-Sensor Fusion-Based Mobile Manipulator Remote Control for Intelligent Smart Home Assistance

TL;DR

The paper presents a wearable, MEMS-based multi-sensor fusion system to remotely control a mobile manipulator for smart-home assistance. It combines MEMS capacitive microphone arrays, IMU, vibration feedback, and pressure sensing with a CNN-LSTM classifier to translate forearm activity into motion and grasp commands. In real-world tests, offline gesture–force classification accuracy reached 88.33% and real-time performance averaged around 1.2 s, with navigation and grasping achieving high success (98% task success, 3.6 cm trajectory deviation) and object transfer showing strong grip/transfer rates (93.3% and 95.6%, respectively, with 91.1% full-task success). These results validate the viability of MEMS-based wearable sensing and multi-sensor fusion for intuitive, safe human–robot collaboration in home environments and point toward future enhancements in motion vocabulary, adaptive grip control, and integrated vision for autonomy.

Abstract

This paper proposes a wearable-controlled mobile manipulator system for intelligent smart home assistance, integrating MEMS capacitive microphones, IMU sensors, vibration motors, and pressure feedback to enhance human-robot interaction. The wearable device captures forearm muscle activity and converts it into real-time control signals for mobile manipulation. The wearable device achieves an offline classification accuracy of 88.33\%\ across six distinct movement-force classes for hand gestures by using a CNN-LSTM model, while real-world experiments involving five participants yield a practical accuracy of 83.33\%\ with an average system response time of 1.2 seconds. In Human-Robot synergy in navigation and grasping tasks, the robot achieved a 98\%\ task success rate with an average trajectory deviation of only 3.6 cm. Finally, the wearable-controlled mobile manipulator system achieved a 93.3\%\ gripping success rate, a transfer success of 95.6\%\, and a full-task success rate of 91.1\%\ during object grasping and transfer tests, in which a total of 9 object-texture combinations were evaluated. These three experiments' results validate the effectiveness of MEMS-based wearable sensing combined with multi-sensor fusion for reliable and intuitive control of assistive robots in smart home scenarios.

Figures (8)

• Figure 1: (a) Overview of the main operational flow between the wearable device and the mobile robot. The system uses a switch to select the task, and based on the acquired muscle signals and arm movements, it controls the mobile robot accordingly. This process ensures that commands are transmitted in real time, enabling effective human-robot interaction. (b) Illustration of the mobile robot's movement control. When the user's hand rotates forward or backward, the robot moves in the corresponding direction; similarly, lateral hand tilts induce left or right turns. This intuitive control mechanism allows the robot to navigate with precision in dynamic environments. (c) Depiction of the robot's grasping control. A finger-closing motion triggers the robot to grasp and pick up an object, whereas a finger-opening motion commands the robot to release the object or handle a square/box-like object. This adaptive grasping strategy supports the handling of objects with varying sizes, shapes, and textures.
• Figure 2: (a) The wearable device comprises three main components: a glove section featuring a vibration motor, IMU, and switch; a central module with an Arduino for signal processing; and a rear segment housing multiple MMG sensors for muscle activity detection. (b) Side view of the mobile manipulator, highlighting the manipulator arm equipped with pressure sensors for force-based interaction. (c) Top view of the mobile manipulator, illustrating its front-mounted manipulator section and a rear portion dedicated to motion control signal reception and processing.
• Figure 3: (a) Experimental setup designed for the MEMS microphone used to test different sensors under the same pressure and identical sound frequency conditions. (b) Frequency response spectrum of three MEMS sensors using three kinds of material(Top: Silicone, Middle: Air, Buttom: Hot Glue) after band-pass filtering. (c) Final internal sturcture of the MEMS sensor.
• Figure 4: 3D scatter plot showing the distribution of common household objects based on weight (50-300g), width (cm), and surface roughness (Ra $\mu$m). Each point represents an individual item, providing a visual reference for the variability in these dimensions, aiding in the design and optimization of the robotic manipulator.
• Figure 5: (a) Cross-sectional diagram of the forearm showing the placement of MEMS sensors on five key muscles: Flexor Carpi Radialis, Flexor Carpi Ulnaris, Extensor Digitorum, Extensor Carpi Radialis Longus, and Extensor Carpi Radialis Brevis. The numbered labels indicate the sensor locations relative to each muscle, allowing for targeted signal detection during various hand and wrist movements. (b) The wearable device used for testing consists of MEMS sensors and an Arduino Mega for data acquisition. The sensors are positioned on the forearm to capture muscle activity signals, while the Arduino Mega serves as the data collection and processing unit, ensuring real-time monitoring during different motion tasks. (c) Basic characteristics of muscle signals during different movements, including gripping, finger extension, pinching, flicking, wrist flexion, and wrist extension. Each set of graphs corresponds to the signal responses from the five targeted muscles, illustrating how the amplitude and pattern of muscle activity vary depending on the type of movement performed.