Adaptive Wiping: Adaptive contact-rich manipulation through few-shot imitation learning with Force-Torque feedback and pre-trained object representations

TL;DR

This work tackles the challenge of adapting imitation-learned policies to contact-rich manipulation of deformable objects with limited demonstrations. It introduces a two-stage approach that pre-trains a sponge-property encoder in simulation and then trains a motion-decoder together with an FT-based height controller on real data, enabling closed-loop adaptation to unseen surface heights and sponge properties. Key contributions include the latent sponge representation $Z_{\text{sponge}} \in \mathbb{R}^5$, a time-series FT feedback loop via a TCN-based encoder, and strong empirical results showing near-reference-force performance across diverse sponges and heights, including wall wiping. The method demonstrates practical impact by achieving high adaptability in real-robot wiping tasks, offering a path toward robust, few-shot manipulation of deformable objects, though it reports higher variability than human demonstrations and suggests extending to non-deformable objects and larger pre-training data.

Abstract

Imitation learning offers a pathway for robots to perform repetitive tasks, allowing humans to focus on more engaging and meaningful activities. However, challenges arise from the need for extensive demonstrations and the disparity between training and real-world environments. This paper focuses on contact-rich tasks like wiping with soft and deformable objects, requiring adaptive force control to handle variations in wiping surface height and the sponge's physical properties. To address these challenges, we propose a novel method that integrates real-time force-torque (FT) feedback with pre-trained object representations. This approach allows robots to dynamically adjust to previously unseen changes in surface heights and sponges' physical properties. In real-world experiments, our method achieved 96% accuracy in applying reference forces, significantly outperforming the previous method that lacked an FT feedback loop, which only achieved 4% accuracy. To evaluate the adaptability of our approach, we conducted experiments under different conditions from the training setup, involving 40 scenarios using 10 sponges with varying physical properties and 4 types of wiping surface heights, demonstrating significant improvements in the robot's adaptability by analyzing force trajectories. The video of our work is available at: https://sites.google.com/view/adaptive-wiping

Figures (11)

• Figure 1: Wiping Experiments: Pre-trained sponge properties in simulation (top), collected real-world data via exploratory actions and human demonstrations (middle), and tested 40 scenarios with 10 sponges, including 9 unseen sponges, and 4 surface heights, including a wall (bottom).
• Figure 2: Overview of our proposed framework. First, we pre-train the sponge properties encoder $\phi_{\text{sponge}}$ using simulated unlabeled data (Pre-training step \ref{['subs:pre-training step']}). Then, we train the motion trajectory decoder $\theta_{\text{traj}}$ and the FT feedback loop $\phi_{\text{ft}}-\theta_{\text{height}}$ to obtain the wiping policy with the active inference of applied force using few-shot human demonstration data (Training step \ref{['subs:training step']}). Finally, we deploy the acquired policy on real robot hardware (Deployment \ref{['subs:deployment']}).
• Figure 3: Manipulation processes of 3 different settings (low, high, sloped) using an unseen sponge that was not included in the training data. The right plots show FT profiles. The baseline simply traces the demonstration and reproduces vertical motion without considering setting changes (gray). In contrast, our method adapts to those changes while maintaining the desired wiping motion (red).
• Figure 4: 10 sponges used in the experiments. One ready-made sponge (normal sponge) for training and the deployment and 9 custom-made sponges with different physical properties (3 stiffness levels $\times$ 3 friction levels) as previously unseen sponges for the deployment.
• Figure 5: Experimental results: Baselines and Ours under Various Conditions. The contact percentage indicates the proportion of time steps where force was applied to press a sponge and the number in () represents the ratio of the average force in the z-direction to that of the corresponding demonstrations (reference force) shown in Table \ref{['tab:demonstration']}.