An Open Source Computer Vision and Machine Learning Framework for Affordable Life Science Robotic Automation

Abstract

We present an open-source robotic framework that integrates computer vision and machine learning based inverse kinematics to enable low-cost laboratory automation tasks such as colony picking and liquid handling. The system uses a custom trained U-net model for semantic segmentation of microbial cultures, combined with Mixture Density Network for predicating joint angles of a simple 5-DOF robot arm. We evaluated the framework using a modified robot arm, upgraded with a custom liquid handling end-effector. Experimental results demonstrate the framework's feasibility for precise, repeatable operations, with mean positional error below 1 mm and joint angle prediction errors below 4 degrees and colony detection capabilities with IoU score of 0.537 and Dice coefficient of 0.596.

Figures (4)

• Figure 1: To automate the colony detection and sampling process, we developed a low-cost and open source based robotic platform integrating a modified 5-DOF robot based on the design in dejan2023 chosen for its ease of assembly, readily available parts and affordability. The arm was fitted with upgraded Hitec D645MW and Towerpro MG92B servos for improved precision. At the end-effector, we attached an open-source digital pipette adapted from Yoshikawa et al. yoshikawa2023 with a 10 mL syringe, enabling the precise aspiration and dispenstion of liquids and a high resolution SVPRO 8 MP USB camera. To control the servo motors we used an Arduino Uno microcontroller arduino_uno because of its popularity in robots, expansive documentation, open design, and low cost.
• Figure 2: The Flow chart illustrates the control architecture for the automated colony picking system. The robot first hovers over the Petri dish and captures an image using the camera. The image is passed through the trained U-net segmentation model, if any colonies are detected, the center points of the contours bounding the colonies are extracted and converted from pixel coordinates to world coordinates. These coordinates are passed through the MDN-based IK model to compute the required joint angles. Lastly the robot arm moves to the picking point to acquire the sample. If no colonies are detected, the system outputs a message stating that nothing was found and all samples have been collected.
• Figure 3: Representative samples from the agar plate image dataset used for training and testing the U-net segmentation model. a) A homemade agar plate using beef bouillon, sugar and gelatin, which produced slightly clear areas along the outer edge due to evaporation of some of the water used in production. b) A standard laboratory prepared agar plate using conventional techniques and materials, resulting in much cleaner and richer bacterial colony growth.
• Figure 4: Example outputs from the semantic segmentation model on two agar plate images. (Left): Original Input image of an agar plate; (Right) Corresponding predicted segmentation mask. The green circles indicate the center point of the detected colonies and indicate the input used for the downstream IK calculations. a). Homemade agar plate prepared using beef bouillon, sugar and gelatin. b). Standard laboratory prepared agar plate.