ManiDreams: An Open-Source Library for Robust Object Manipulation via Uncertainty-aware Task-specific Intuitive Physics

Abstract

Dynamics models, whether simulators or learned world models, have long been central to robotic manipulation, but most focus on minimizing prediction error rather than confronting a more fundamental challenge: real-world manipulation is inherently uncertain. We argue that robust manipulation under uncertainty is fundamentally an integration problem: uncertainties must be represented, propagated, and constrained within the planning loop, not merely suppressed during training. We present and open-source ManiDreams, a modular framework for uncertainty-aware manipulation planning over intuitive physics models. It realizes this integration through composable abstractions for distributional state representation, backend-agnostic dynamics prediction, and declarative constraint specification for action optimization. The framework explicitly addresses three sources of uncertainty: perceptual, parametric, and structural. It wraps any base policy with a sample-predict-constrain loop that evaluates candidate actions against distributional outcomes, adding robustness without retraining. Experiments on ManiSkill tasks show that ManiDreams maintains robust performance under various perturbations where the RL baseline degrades significantly. Runnable examples on pushing, picking, catching, and real-world deployment demonstrate flexibility across different policies, optimizers, physics backends, and executors. The framework is publicly available at https://github.com/Rice-RobotPI-Lab/ManiDreams

Figures (10)

• Figure 1: ManiDreams unifies perceptual, parametric, and structural uncertainties through Task-specific Intuitive Physics (TSIP) and Domain-randomized Instance Set (DRIS). Left: Simulation-based TSIP across standard tasks (PushCube, PickCube, PushT), where the overlaid objects are DRIS instances. Right: A card picking task and a pushing task driven by simulation-based and learning-based TSIPs.
• Figure 2: Three-layer architecture. Layer 1 defines abstract interfaces for the four core abstractions, analogous to OMPL's StateSpace, StatePropagator, StateValidityChecker, and Planner. Layer 2 provides concrete implementations unified behind a single environment class. Layer 3 is user-facing: adding a new task requires only populating this layer. Dashed arrows indicate inheritance.
• Figure 3: Runtime pipeline of ManiDreams. The planning loop (yellow): The solver generates candidate actions, predicts their DRIS outcomes via TSIP, and selects the best action satisfying the caging constraint; the loop may iterate for refinement. The execution loop (green): The selected action is appended to an action chunk; once complete, the chunk is dispatched to the real robot or simulator, which observes the outcome and produces an updated DRIS for the next planning cycle.
• Figure 4: Minimal working example. Three steps configure and launch the full sample-predict-constrain pipeline. Setting num_samples=1 and omitting the cage reduces the pipeline to direct policy execution, serving as the baseline in all experiments.
• Figure 5: Object pushing with circular trajectory following using different TSIPs. Random target objects are selected from the YCB dataset calli2017ycb. Top: Simulation-based TSIP. Bottom: Learning-based TSIP based on a pre-trained diffusion model.