Subspace-wise Hybrid RL for Articulated Object Manipulation

TL;DR

SwRL addresses articulated-object manipulation under unknown dynamics by decomposing the end-effector task space into three subspaces $\mathcal{S}_K$, $\mathcal{S}_G$, and $\mathcal{S}_R$ and learning independent RL policies for $\mathcal{S}_K$-force control and $\mathcal{S}_R$-redundant motion. The framework integrates these policies through a hybrid force/motion controller, enabling adaptive force modulation and exploitation of redundancy to improve dexterity and sample efficiency. Empirical results in simulation and real-world valve, door, and drawer tasks show SwRL outperforms baselines and approaches planning methods like CBiRRT while delivering real-time performance and robustness to unknown dynamics. The work highlights the value of subspace-wise learning for contact-rich manipulation and motivates future automatic decomposition for broader object classes.

Abstract

Articulated object manipulation is a challenging task, requiring constrained motion and adaptive control to handle the unknown dynamics of the manipulated objects. While reinforcement learning (RL) has been widely employed to tackle various scenarios and types of articulated objects, the complexity of these tasks, stemming from multiple intertwined objectives makes learning a control policy in the full task space highly difficult. To address this issue, we propose a Subspace-wise hybrid RL (SwRL) framework that learns policies for each divided task space, or subspace, based on independent objectives. This approach enables adaptive force modulation to accommodate the unknown dynamics of objects. Additionally, it effectively leverages the previously underlooked redundant subspace, thereby maximizing the robot's dexterity. Our method enhances both learning efficiency and task execution performance, as validated through simulations and real-world experiments. Supplementary video is available at https://youtu.be/PkNxv0P8Atk

Figures (10)

• Figure 1: Illustration of motion directions for prismatic and revolute joints. (a) Prismatic Joint: The motion is linear along the $\hat{Z}_o$ axis, with velocity $v$. (b) Revolute Joint: The motion is rotational around the $\hat{Z}_o$ axis, with angular velocity $\omega$.
• Figure 2: Illustration of object-oriented frame of various objects.
• Figure 3: Overview of the Subspace-wise hybrid RL (SwRL) framework for articulated object manipulation. Task space decomposition is performed in the object-oriented frame $\{O\}$, dividing the task space into three subspaces: the kinematic subspace $\mathcal{S}_K$, geometric subspace $\mathcal{S}_G$, and redundant subspace $\mathcal{S}_R$. The kinematic force control policy generates the magnitude of force vector within the kinematic subspace $\mathcal{S}_K$, while the redundant motion control policy produces motion within the redundant subspace $\mathcal{S}_R$. Both policies operate in parallel, with their outputs integrated into the hybrid force/motion controller.
• Figure 4: Illustration of geometric constraints for a handwheel valve. The grey circle depicts the handle or the circular trace on the object, while the black object represents the end-effector. The end-effector cannot rotate in the yaw direction from its pose when grasping the object.
• Figure 5: Conceptual illustration of door opening motion with and without redundant space motion. The maximum door opening angle, $\theta_{max}$, is shown for the case without redundant space motion (left) and with redundant motion (right). The right case demonstrates a larger $\theta_{max}$.