IFOR: Iterative Flow Minimization for Robotic Object Rearrangement

TL;DR

IFOR tackles the problem of rearranging unseen objects from RGB-D goal scenes by framing it as iterative flow minimization between the current and goal images. It relies on a rearrangement-trained RAFT to predict large-displacement optical flow, combines this with unseen-object segmentation and a RANSAC-based 3D pose estimation to recover per-object 6-DoF transforms, and then plans collision-free pick-and-place actions. The approach is trained entirely in simulation and transfers to real-world robots without fine-tuning, outperforming prior methods in scenarios requiring orientation changes. Key contributions include a task-specific optical flow model, a zero-shot segmentation and pose-estimation pipeline, and a planning-execution loop with robust collision checking, enabling practical rearrangement in cluttered, unknown environments.

Abstract

Accurate object rearrangement from vision is a crucial problem for a wide variety of real-world robotics applications in unstructured environments. We propose IFOR, Iterative Flow Minimization for Robotic Object Rearrangement, an end-to-end method for the challenging problem of object rearrangement for unknown objects given an RGBD image of the original and final scenes. First, we learn an optical flow model based on RAFT to estimate the relative transformation of the objects purely from synthetic data. This flow is then used in an iterative minimization algorithm to achieve accurate positioning of previously unseen objects. Crucially, we show that our method applies to cluttered scenes, and in the real world, while training only on synthetic data. Videos are available at https://imankgoyal.github.io/ifor.html.

Figures (7)

• Figure 1: Overview of the IFOR algorithm. IFOR takes as input RGB+D images of a current and a goal scene, and uses these to make predicts as to which objects should move and by which transformations, using RAFT to estimate optical flow. This is then sent to a robot planning and execution pipeline which is capable of grasping of unknown objects and motion planning in scenes with unknown geometry.
• Figure 2: Examples from the synthetic dataset used for training RAFT. Images come in pairs of initial and final images, each containing assorted objects in clutter. Between the two images, there is a large, randomly-sampled transformations. Retraining RAFT on these large discontinuities is essential to our approach.
• Figure 3: Examples of different real-world scenes warped after object points have been transformed according to poses estimated by IFOR. Poses are estimated according to predicted optical flow features.
• Figure 4: Two examples comparing the qualitative performance of NeRP qureshi:rss2021 to IFOR. In both of these examples, IFOR both more closely matches the goal image, and matches the orientation much more precisely than NeRP does.
• Figure 5: User scores for IFOR vs. NeRP qureshi:rss2021. When asked to rate performance of the two methods on a scale of 1-4, users preferred IFOR by a wide margin. Users chose IFOR over NeRP in almost all situations, when looking at either position only (94%) or full pose (position and orientation, 92%).