Real2Sim based on Active Perception with automatically VLM-generated Behavior Trees

TL;DR

This work addresses the challenge of building physically accurate simulations from real environments by enabling autonomous, intent-driven Real2Sim. It introduces an active perception framework that uses vision-language models to infer the minimal set of physical parameters needed for a given simulation goal and generates executable Behavior Trees over a library of atomic actions to acquire those parameters through compliant contact-rich interactions on a Franka Panda. The approach tightly couples high-level reasoning with physically grounded robot actions, producing a physics-aware MuJoCo replica without task-specific templates or expert-designed exploration. Experiments demonstrate accurate estimation of object mass, surface height, and friction-related quantities under occlusion and incomplete prior models, illustrating interpretable and adaptable simulation construction with potential applications in robotics and learning. Overall, the paper advances autonomous, intent-aware Real2Sim pipelines by integrating multi-modal reasoning, BT-based planning, and real-world robotic interaction toward scalable digital twins.

Abstract

Constructing an accurate simulation model of real-world environments requires reliable estimation of physical parameters such as mass, geometry, friction, and contact surfaces. Traditional real-to-simulation (Real2Sim) pipelines rely on manual measurements or fixed, pre-programmed exploration routines, which limit their adaptability to varying tasks and user intents. This paper presents a Real2Sim framework that autonomously generates and executes Behavior Trees for task-specific physical interactions to acquire only the parameters required for a given simulation objective, without relying on pre-defined task templates or expert-designed exploration routines. Given a high-level user request, an incomplete simulation description, and an RGB observation of the scene, a vision-language model performs multi-modal reasoning to identify relevant objects, infer required physical parameters, and generate a structured Behavior Tree composed of elementary robotic actions. The resulting behavior is executed on a torque-controlled Franka Emika Panda, enabling compliant, contact-rich interactions for parameter estimation. The acquired measurements are used to automatically construct a physics-aware simulation. Experimental results on the real manipulator demonstrate estimation of object mass, surface height, and friction-related quantities across multiple scenarios, including occluded objects and incomplete prior models. The proposed approach enables interpretable, intent-driven, and autonomously Real2Sim pipelines, bridging high-level reasoning with physically-grounded robotic interaction.

Figures (8)

• Figure 1: Compact Real2Sim adaptive framework. The user specifies the desired objective, while the system prompt (fixed and not accessible to the user) explains how to deal with available atomic functions and the desired output. Then the VLM returns the Behavior Tree for the acquisition of the missing parameters. Once all physical parameters are available, a physics-aware replica of the environment is built in MuJoCo.
• Figure 2: Example of BT generated for the estimation of table height and mass of the blue bottle. First, the robot executes the sub-tree to acquire the table height, then it executes the other to acquire the bottle mass. As it is possible to see from the second subtree, the VLM is instructed to approach objects before moving to their poses for picking. Note that each sub-tree is built with a composition of atomic actions from set $\mathcal{A}$.
• Figure 3: First scenario sequence of parameters estimation. The robot first touches the table to acquire its height with the end-effector pose. Then it lifts the bottle to acquire that mass with torque sensors and then puts the object in its original position.
• Figure 4: BT generated for bottles mass acquisition only, when $\mathcal{D}$ is available, and table height is known.
• Figure 5: Real environment $\mathcal{I}$ picture from camera (left) and simulation $\mathcal{I}$ of it (right). When the simulation image is used as a prompt, only a graphical representation with bottle meshes is used.