Vegetable Peeling: A Case Study in Constrained Dexterous Manipulation

TL;DR

The paper tackles constrained dexterous manipulation for food peeling by learning a stop-capable reorientation controller using a teacher–student RL framework. A goal-conditioned teacher learns to reorient objects along a fixed axis and stop, guided by a simplified reward and a stop signal, while a student imitates the teacher using only proprioception and a Transformer architecture to enable real-world deployment. Key contributions include a one-step demonstration reward, a time-sustained success criterion with $C_{ori}$ and $C_{contact}$, reset constraints to enforce feasibility, and action interpolation for smooth real-time control, plus dual peeling pipelines (teleoperation and vision-based) that demonstrate downstream peeling. The approach enables robust, in-hand reorientation of diverse vegetables and secure grasping to support peeling, with real-world tests showing meaningful rotation, timely stopping, and high grasp success, albeit with challenges in small objects and perception reliability. This work advances constrained dexterous manipulation toward practical downstream tasks and lays groundwork for integrating richer perception and autonomous peeling strategies.

Abstract

Recent studies have made significant progress in addressing dexterous manipulation problems, particularly in in-hand object reorientation. However, there are few existing works that explore the potential utilization of developed dexterous manipulation controllers for downstream tasks. In this study, we focus on constrained dexterous manipulation for food peeling. Food peeling presents various constraints on the reorientation controller, such as the requirement for the hand to securely hold the object after reorientation for peeling. We propose a simple system for learning a reorientation controller that facilitates the subsequent peeling task. Videos are available at: https://taochenshh.github.io/projects/veg-peeling.

Figures (10)

• Figure 1: We present a dexterous manipulation system that utilizes an Allegro hand mounted on a Franka robot arm to reorient food items for downstream peeling. The other Franka robot arm (the right arm in the figure) uses its gripper to grasp a peeler for peeling. The reorientation controller for the Allegro hand is learned through reinforcement learning, while the peeling is performed via teleoperation. In the figure, we demonstrate the process of reorienting and peeling a melon, a sweet potato, and a squash from top to bottom row.
• Figure 2: Robot setup for reorientation and peeling.
• Figure 3: (a) shows an example of the rotational axis of a melon. (b) shows an example where the object's orientation (the blue line) has a large deviation from the desired rotational axis (the green line). We reset the episode when this occurs. (c) shows the policy Architecture for the teacher and the student. In this figure, we use $\bm{o}_t$ to represent all the policy input at each time step.
• Figure 4: Examples of joint position commands after interpolation sent to a low-level PD controller. represents the actual joint position of the motor. is the computed desired joint position. on the green line shows the interpolated joint position commands that are sent to the low-level PD controller. (a) shows the case of $\bm{q}_{t+1}^{cmd}=\bm{q}_t+\bm{a}_t$, while (b) shows the case of $\bm{q}_{t+1}^{cmd}=\bm{q}_t^{cmd}+\bm{a}_t$. We can see that (b) generates much smoother joint commands.
• Figure 5: (a): the Allegro hand holds a papaya to be peeled. (b): we utilize Grounded SAM to segment the papaya. (c): the 3D point cloud representing the segmented papaya's exposed surface. (d): we take a slice of this point cloud at the center region along the papaya's longest axis. (e): the points within this center slice are projected onto the central plane aligned with the axis. (f): we fit a spline curve to the projected points to obtain the desired trajectory for the peeler tip to follow.