Vision-based robot manipulation of transparent liquid containers in a laboratory setting

TL;DR

A vision-based system for liquid volume estimation and a simulation-driven pouring method particularly designed for containers with small openings are introduced, followed by an applied real-world integration of cell culture automation using a UR5 robotic arm.

Abstract

Laboratory processes involving small volumes of solutions and active ingredients are often performed manually due to challenges in automation, such as high initial costs, semi-structured environments and protocol variability. In this work, we develop a flexible and cost-effective approach to address this gap by introducing a vision-based system for liquid volume estimation and a simulation-driven pouring method particularly designed for containers with small openings. We evaluate both components individually, followed by an applied real-world integration of cell culture automation using a UR5 robotic arm. Our work is fully reproducible: we share our code at at \url{https://github.com/DaniSchober/LabLiquidVision} and the newly introduced dataset LabLiquidVolume is available at https://data.dtu.dk/articles/dataset/LabLiquidVision/25103102.

Figures (13)

• Figure 1: We present a robotic setup for cell culture automation using exclusively existing laboratory equipment. (a) First, we propose a two-step model architecture to estimate the liquid volume in transparent laboratory containers using a single RGB image. (b) Secondly, from a large pool of simulated pouring trajectories, the simulation that approximates the real-world state best is performed on the UR5. Hereby, the pouring strategy is constrained to rotate around the liquid exit point which is crucial for receiving containers with small openings such as cell culture flasks.
• Figure 2: Visualization of the issue with pouring movements only considering a wrist rotation. Left: Successful pouring only using rotation because of a receiving container with a large opening surface and a pouring container with a small height. Right: Liquid is spilled when only rotating a cell culture flask around the TCP.
• Figure 3: Visualization of the pouring movement for two exemplary laboratory containers. $\theta$ represents the pouring angle, $\beta$ the angle between the $TCP$ and the $CoR$ of the bottle in the horizontal position, $\alpha$ the rotation of the bottle to the same angle as the receiving flask ($\alpha=14.5°$), and $\alpha_{start}$ the start angle between the $TCP$ and the $CoR$. Left: Pouring with the cell culture flask. Center: Explanation of the start position of the bottle. Right: Pouring with the media/washing solution bottle.
• Figure 4: Overview of the system prototype for cell culture automation. (1) Automated incubator. (2) Microscope. (3) Trypsin unit. (4) Capper/Decapper. (5) Lid holders. (6) Flask storage. (7) Flask holder for pouring. (8) UR5e with 3D-printed gripper fingers. (9) Heating and cooling of liquids unit.
• Figure 5: An exemplary simulation scene of the media bottle executing the new pouring movement.