Closed-Loop Robotic Manipulation of Transparent Substrates for Self-Driving Laboratories using Deep Learning Micro-Error Correction

TL;DR

The paper tackles the bottleneck of autonomous handling of fragile, transparent substrates in self-driving laboratories. It introduces ASHE, a closed-loop system that integrates a 5-DOF robotic arm with a deformable gripper, a dual-actuated substrate dispenser, and a fused geometric–deep learning vision detector using lateral blue illumination to verify placement. The key contributions are a robust macro- and micro-scale placement error detection framework and demonstration of high reliability, achieving 98.5% first-time placement accuracy across 130 reloads with automatic correction for misplacements. This work significantly enhances SDL automation by reliably reloading substrates, reducing downtime, and enabling faster, unattended materials discovery workflows.

Abstract

Self-driving laboratories (SDLs) have accelerated the throughput and automation capabilities for discovering and improving chemistries and materials. Although these SDLs have automated many of the steps required to conduct chemical and materials experiments, a commonly overlooked step in the automation pipeline is the handling and reloading of substrates used to transfer or deposit materials onto for downstream characterization. Here, we develop a closed-loop method of Automated Substrate Handling and Exchange (ASHE) using robotics, dual-actuated dispensers, and deep learning-driven computer vision to detect and correct errors in the manipulation of fragile and transparent substrates for SDLs. Using ASHE, we demonstrate a 98.5% first-time placement accuracy across 130 independent trials of reloading transparent glass substrates into an SDL, where only two substrate misplacements occurred and were successfully detected as errors and automatically corrected. Through the development of more accurate and reliable methods for handling various types of substrates, we move toward an improvement in the automation capabilities of self-driving laboratories, furthering the acceleration of novel chemical and materials discoveries.

Figures (8)

• Figure 1: The Automated Substrate Handling and Exchange (ASHE) system for robotic manipulation and error correction of transparent substrate placement. ASHE combines the use of a dual-actuated dispenser, a 5-degree-of-freedom robotic arm with a specialized deformable gripper, and a deep learning computer vision detector to automate substrate reloading and unloading from experimental equipment. Through vision-based detection and automated correction of micro-scale errors in the placement of a substrate, ASHE carefully manipulates fragile and transparent substrates with high accuracy and repeatability, suited for the long, autonomous experimental campaigns of self-driving laboratories.
• Figure 2: Robotic path plan for substrate reloading and grasping using specialized gripper fingers. The robotic arm path plan first (a) removes used substrates from the SDL transporter and moves them to a waste bin, then (b) picks up a fresh substrate from the dispenser, and finally (c) places the fresh substrate onto the SDL transporter. (d) Fragile substrates are manipulated with care, avoiding fracture through the use of a dual-material finger design that employs both rigid polylactic acid (PLA) and deformable thermoplastic polyurethane (TPU). Four points of contact are made between the substrate and the deformable TPU to ensure a secure grasp during robot motion.
• Figure 3: Dual-actuated dispenser sets fresh substrates for the robotic arm to grasp and move. (a) A single substrate is pushed from the bottom of a stack using horizontal actuation. (b) At the horizontal actuator's full extension, a vertical actuator is released to block the substrate from reentry. (c) The substrate stays in place as the horizontal actuator retracts. (d) The vertical actuator retracts, leaving the substrate in place for the robotic arm to grasp.
• Figure 4: Lateral illumination of transparent substrates using blue light for computer vision detection. (a) Rendering of a transparent glass substrate in ambient light with no lateral illumination. (b) Rendering of a transparent glass substrate in the dark with blue light lateral illumination. The light refracts within the glass and emits along the edge faces of the substrate, illuminating them for clearer computer vision detection. (c) The computer vision imaging setup with RealSense D435 camera and lateral blue light illumination used for error detection in transparent substrate placement. Experimental images are shown of a transparent glass substrate successfully and unsuccessfully placed into the target slot of the transporter under the lateral illumination condition.
• Figure 5: Fused geometric model (GM) and convolutional neural network (CNN) decision tree for reliable classification of placement errors for transparent substrates. A successful placement is declared only if the GM and CNN both output success classifications of the substrate placement.