Towards Generalizable Robotic Data Flywheel: High-Dimensional Factorization and Composition

Abstract

The lack of sufficiently diverse data, coupled with limited data efficiency, remains a major bottleneck for generalist robotic models, yet systematic strategies for collecting and curating such data are not fully explored. Task diversity arises from implicit factors that are sparsely distributed across multiple dimensions and are difficult to define explicitly. To address this challenge, we propose F-ACIL, a heuristic factor-aware compositional iterative learning framework that enables structured data factorization and promotes compositional generalization. F-ACIL decomposes the data distribution into structured factor spaces such as object, action, and environment. Based on the factorized formulation, we develop a factor-wise data collection and an iterative training paradigm that promotes compositional generalization over the high-dimensional factor space, leading to more effective utilization of real-world robotic demonstrations. With extensive real-world experiments, we show that F-ACIL can achieve more than 45% performance gains with 5-10$\times$ fewer demonstrations comparing to that of which without the strategy. The results suggest that structured factorization offers a practical pathway toward efficient compositional generalization in real-world robotic learning. We believe F-ACIL can inspire more systematic research on building generalizable robotic data flywheel strategies. More demonstrations can be found at: https://f-acil.github.io/

Figures (9)

• Figure 1: Illustration of data distribution with specific properties. (1)The 3D surfaces (blue-purple) represent different training data distributions, while the contour maps(warm amber) show the shared evaluation distribution. (2)We compare three data distribution: (a) a narrow Gaussian-like distribution with limited coverage, (b) a quasi-uniform distribution with full coverage but low efficiency, and (c) F-ACIL with multiple gaussian modes, which achieves efficient, broad coverage via factor-wise composition.
• Figure 2: Overview. (1)F-ACIL explicitly decomposes robotic manipulation into three factors: Object, Action, and Environment. This factorized representation is introduced to systematically expand the distribution of training data in real-world robotic learning. (2)F-ACIL with factorized representation allows principled compositional generalization to out-of-distribution (OOD) instances.
• Figure 3: Capabilities. In both Pick-and-Place and Open-and-Close, all cases exhibit robust compositional generalization across out-of-domain objects, actions, and environments.
• Figure 4: Demonstrations collected for iteration 1 under $\mathcal{O}$ space The initial demonstrations are sampled along the main diagonal of the factor space, with an equal number of samples per composition. The background color encodes the success rate(SR). Black dots mark the compositions included in the dataset, denoted as $D_1$.
• Figure 5: Iterative Search Procedure in Object Space In each iteration, different colors indicate the performance under specific factor compositions. The background color encoding follows the same scheme as in Fig. \ref{['fig:iter1']}. Points highlighted in blue mark factor compositions that are newly added into the demonstrations in the current iteration. The number of points reflects the distribution of ratio across different factor compositions.