MATTERIX: toward a digital twin for robotics-assisted chemistry laboratory automation

TL;DR

MATTERIX presents a GPU-accelerated digital twin framework that fuses robotic manipulation, particle dynamics, heat transfer, and approximate chemical kinetics to accelerate chemistry workflow development. By integrating a modular semantics engine with NVIDIA Isaac Sim and a FoundationPose-based perception system, it enables multi-scale simulations and sim-to-real deployment across heterogeneous lab robots and devices. The approach delivers an extensive open-source asset library, a hierarchical skill library, and a real-to-sim pipeline validated on pick-and-place, pouring, and liquid-handling tasks, while highlighting limitations in reaction modeling and long-horizon planning. Overall, MATTERIX offers a scalable platform for rapid prototyping and testing of automated chemistry workflows, with potential to reduce real-world experimentation, enable data-driven workflow optimization, and support AI-driven experimental planning in silico.

Abstract

Accelerated materials discovery is critical for addressing global challenges. However, developing new laboratory workflows relies heavily on real-world experimental trials, and this can hinder scalability because of the need for numerous physical make-and-test iterations. Here we present MATTERIX, a multiscale, graphics processing unit-accelerated robotic simulation framework designed to create high-fidelity digital twins of chemistry laboratories, thus accelerating workflow development. This multiscale digital twin simulates robotic physical manipulation, powder and liquid dynamics, device functionalities, heat transfer and basic chemical reaction kinetics. This is enabled by integrating realistic physics simulation and photorealistic rendering with a modular graphics processing unit-accelerated semantics engine, which models logical states and continuous behaviors to simulate chemistry workflows across different levels of abstraction. MATTERIX streamlines the creation of digital twin environments through open-source asset libraries and interfaces, while enabling flexible workflow design via hierarchical plan definition and a modular skill library that incorporates learning-based methods. Our approach demonstrates sim-to-real transfer in robotic chemistry setups, reducing reliance on costly real-world experiments and enabling the testing of hypothetical automated workflows in silico. The project website is available at https://accelerationconsortium.github.io/Matterix/ .

Figures (10)

• Figure 1: MatteriX architecture and components.MatteriX generates a digital twin environment by specifying objects and their poses using a library of digital and photorealistic assets (1). It solves a given workflow to perform the chemistry experiment (2). To perform the experiment, it trains new policies that are not available in the skill library but are required for experimenting. A multi-step workflow is generated by a human or large language model. Using a database of known reactions and chemical kinetics approximation, MatteriX verifies if the target chemical is synthesized. MatteriX is deployed in the real world and performs a chemistry experiment (3). The object poses are perceived using a camera stream in the real environment, and the simulated robots are matched with the real robots.
• Figure 2: Digital twin requirements and comparison with MatteriX. The top figure illustrates key requirements and benefits of digital twins in chemistry lab automation, while the bottom compares MatteriX with related works.
• Figure 3: Examples of MatteriX digital twin full chemistry laboratory environment and its workstations. A chemistry lab simulation framework comprising different (mobile) robotic chemists, instruments, and glassware across various laboratory tasks, including materials and fluids (A-F). (G) demonstrates how an environment is created.
• Figure 4: A library of manipulation, perception skills, and device functionalities in MatteriX. The figure demonstrates the behaviour and performance of our integrated physics and semantics engine. (A) illustrates the creation of abstract skills using a hierarchical state machine. (B–F) showcase various skills for manipulating rigid and granular objects. (I) presents object pose estimation via computer vision. (J–K) highlight liquid-handling device functionalities enabled by the integrated semantics engine. (L) compares simulation throughput for a heat transfer environment with and without the semantics engine. For 2048 and 4096 environments, the GPU overhead with the semantics engine results in FPS (Frames Per Second) reductions of only 2.97% and 3.42%, respectively, compared to the physics-only engine.
• Figure 5: Multi-scale simulation of chemistry experiments. (A) A single-step organic chemistry experiment demonstrating the interaction of physical manipulation, heat transfer, and chemical kinetics simulation. (B) A two-step reduction-oxidation chemistry experiment illustrating the interaction of physical manipulation and chemical kinetics simulation. See \ref{['sec:appendix:Deployment-Digital-Twin-Workflows-Real-Setup']} for the video of experiment (B).