Control of Microrobots with Reinforcement Learning under On-Device Compute Constraints

TL;DR

This work tackles the problem of autonomous, on-device locomotion for sub-centimeter microrobots under strict compute and power constraints. It trains a compact FP32 MLP policy in massively parallel simulation with domain randomization and validates deployment on the SCμM-3C Cortex-M0 SoC, including a hardware-aware analysis of update frequency and quantization. A key contribution is quantized, per-feature Int8 inference that substantially increases update rates while retaining locomotion performance, along with a resource-aware gait selection framework that chooses the optimal gait under hardware budgets. The results show robust, gait-conditioned behavior in simulation and qualitative macro-scale deployment, highlighting practical design insights for edge RL on microcontrollers and paving the way for real microrobot realizations with hardware-aware control policies.

Abstract

An important function of autonomous microrobots is the ability to perform robust movement over terrain. This paper explores an edge ML approach to microrobot locomotion, allowing for on-device, lower latency control under compute, memory, and power constraints. This paper explores the locomotion of a sub-centimeter quadrupedal microrobot via reinforcement learning (RL) and deploys the resulting controller on an ultra-small system-on-chip (SoC), SC$μ$M-3C, featuring an ARM Cortex-M0 microcontroller running at 5 MHz. We train a compact FP32 multilayer perceptron (MLP) policy with two hidden layers ($[128, 64]$) in a massively parallel GPU simulation and enhance robustness by utilizing domain randomization over simulation parameters. We then study integer (Int8) quantization (per-tensor and per-feature) to allow for higher inference update rates on our resource-limited hardware, and we connect hardware power budgets to achievable update frequency via a cycles-per-update model for inference on our Cortex-M0. We propose a resource-aware gait scheduling viewpoint: given a device power budget, we can select the gait mode (trot/intermediate/gallop) that maximizes expected RL reward at a corresponding feasible update frequency. Finally, we deploy our MLP policy on a real-world large-scale robot on uneven terrain, qualitatively noting that domain-randomized training can improve out-of-distribution stability. We do not claim real-world large-robot empirical zero-shot transfer in this work.

Figures (10)

• Figure 1: Single-Chip Micro Mote, SC$\mu$M (left) and SOI-based microrobots (right).
• Figure 2: SOI MEMS Hexapod Walker: legs, inchworm motor, and actuator configuration.
• Figure 3: Quadruped configuration in Simulink Multibody.
• Figure 4: Training pipeline: PhysX rollouts in Isaac Gym $\rightarrow$ PPO update $\rightarrow$ compact MLP policy.
• Figure 5: Evaluation/inference pipeline: Simulink plant on host PC $\leftrightarrow$ UART $\leftrightarrow$ SC$\mu$M (inference).