Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WheelArm-Sim: A Manipulation and Navigation Combined Multimodal Synthetic Data Generation Simulator for Unified Control in Assistive Robotics

Guangping Liu, Tipu Sultan, Vittorio Di Giorgio, Nick Hawkins, Flavio Esposito, Madi Babaiasl

TL;DR

WheelArm-Sim presents a physics-based simulation framework in Isaac Sim that enables integrated wheelchair and robotic arm data collection for assistive robotics. The authors develop a ROS2-driven, human-in-the-loop workflow to gather a multimodal dataset (13 tasks, 232 trajectories, 67,783 samples) including instructions, RGB-D, IMU, and joint states, and validate a baseline LSTM model for action prediction on a mustard-picking task. The work demonstrates the feasibility of data-driven integrated control for WheelArm and identifies key challenges such as dataset size and sim-to-real transfer, guiding future VLA-based approaches and improved user interfaces. Overall, this work provides a foundation for scalable multimodal data collection and learning for unified WheelArm control with potential impact on ADL-assistive robotics.

Abstract

Wheelchairs and robotic arms enhance independent living by assisting individuals with upper-body and mobility limitations in their activities of daily living (ADLs). Although recent advancements in assistive robotics have focused on Wheelchair-Mounted Robotic Arms (WMRAs) and wheelchairs separately, integrated and unified control of the combination using machine learning models remains largely underexplored. To fill this gap, we introduce the concept of WheelArm, an integrated cyber-physical system (CPS) that combines wheelchair and robotic arm controls. Data collection is the first step toward developing WheelArm models. In this paper, we present WheelArm-Sim, a simulation framework developed in Isaac Sim for synthetic data collection. We evaluate its capability by collecting a manipulation and navigation combined multimodal dataset, comprising 13 tasks, 232 trajectories, and 67,783 samples. To demonstrate the potential of the WheelArm dataset, we implement a baseline model for action prediction in the mustard-picking task. The results illustrate that data collected from WheelArm-Sim is feasible for a data-driven machine learning model for integrated control.

WheelArm-Sim: A Manipulation and Navigation Combined Multimodal Synthetic Data Generation Simulator for Unified Control in Assistive Robotics

TL;DR

WheelArm-Sim presents a physics-based simulation framework in Isaac Sim that enables integrated wheelchair and robotic arm data collection for assistive robotics. The authors develop a ROS2-driven, human-in-the-loop workflow to gather a multimodal dataset (13 tasks, 232 trajectories, 67,783 samples) including instructions, RGB-D, IMU, and joint states, and validate a baseline LSTM model for action prediction on a mustard-picking task. The work demonstrates the feasibility of data-driven integrated control for WheelArm and identifies key challenges such as dataset size and sim-to-real transfer, guiding future VLA-based approaches and improved user interfaces. Overall, this work provides a foundation for scalable multimodal data collection and learning for unified WheelArm control with potential impact on ADL-assistive robotics.

Abstract

Wheelchairs and robotic arms enhance independent living by assisting individuals with upper-body and mobility limitations in their activities of daily living (ADLs). Although recent advancements in assistive robotics have focused on Wheelchair-Mounted Robotic Arms (WMRAs) and wheelchairs separately, integrated and unified control of the combination using machine learning models remains largely underexplored. To fill this gap, we introduce the concept of WheelArm, an integrated cyber-physical system (CPS) that combines wheelchair and robotic arm controls. Data collection is the first step toward developing WheelArm models. In this paper, we present WheelArm-Sim, a simulation framework developed in Isaac Sim for synthetic data collection. We evaluate its capability by collecting a manipulation and navigation combined multimodal dataset, comprising 13 tasks, 232 trajectories, and 67,783 samples. To demonstrate the potential of the WheelArm dataset, we implement a baseline model for action prediction in the mustard-picking task. The results illustrate that data collected from WheelArm-Sim is feasible for a data-driven machine learning model for integrated control.
Paper Structure (25 sections, 9 figures, 5 tables, 1 algorithm)

This paper contains 25 sections, 9 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Assistive Robotics
  4. Simulation
  5. Datasets
  6. Activities and Tasks
  7. WHEELARM SIMULATOR
  8. Simulation Scenario
  9. WheelArm Simulation Model
  10. Computation Device
  11. DATA COLLECTION
  12. Data Collection Workflow
  13. Inverse Kinematics Method
  14. DATASET STRUCTURE
  15. BASELINE
  16. ...and 10 more sections

Figures (9)

  • Figure 1: WheelArm-Sim features a human-in-the-loop workflow where teleoperation guides task execution in a physics-based simulator, with real-time data collection to record human instructions, RGB-D images, and robot data.
  • Figure 2: Four areas for tasks in the WheelArm-Sim environment.
  • Figure 3: (a) Isaac Sim built-in objects, (b) self-created objects, and (c,d) online objects. Our dataset includes diverse object types, including deformable materials.
  • Figure 4: ROS2-based Data Collection Workflow integrating user teleoperation, robotic control, and sensor data processing. This workflow enables efficient multimodal data collection for the WheelArms.
  • Figure 5: Dataset Distribution
  • ...and 4 more figures