Coarse-to-Fine Learning for Multi-Pipette Localisation in Robot-Assisted In Vivo Patch-Clamp

TL;DR

This work tackles automated, real-time localisation of multiple pipettes in in vivo patch-clamp experiments using two-photon imaging. A heatmap-augmented coarse-to-fine pipeline integrates CycleGAN-based background elimination, a Vision Transformer encoder for coarse pipette heatmaps, and a ResNet-18-based fine regression to predict precise tip coordinates, with Hungarian matching guiding robust training. The method achieves high localization accuracy, reporting > $98\%$ within $10\,\mu$m and > $89\%$ within $5\,\mu$m, with an average MSE of $2.52\,\mu$m, and operates in real time across CPU and GPU hardware. This approach substantially enhances automation and scalability for robot-assisted in vivo multi-pipette patch-clamp, enabling more efficient investigations of cellular interactions in intact brains.

Abstract

In vivo image-guided multi-pipette patch-clamp is essential for studying cellular interactions and network dynamics in neuroscience. However, current procedures mainly rely on manual expertise, which limits accessibility and scalability. Robotic automation presents a promising solution, but achieving precise real-time detection of multiple pipettes remains a challenge. Existing methods focus on ex vivo experiments or single pipette use, making them inadequate for in vivo multi-pipette scenarios. To address these challenges, we propose a heatmap-augmented coarse-to-fine learning technique to facilitate multi-pipette real-time localisation for robot-assisted in vivo patch-clamp. More specifically, we introduce a Generative Adversarial Network (GAN)-based module to remove background noise and enhance pipette visibility. We then introduce a two-stage Transformer model that starts with predicting the coarse heatmap of the pipette tips, followed by the fine-grained coordination regression module for precise tip localisation. To ensure robust training, we use the Hungarian algorithm for optimal matching between the predicted and actual locations of tips. Experimental results demonstrate that our method achieved > 98% accuracy within 10 μm, and > 89% accuracy within 5 μm for the localisation of multi-pipette tips. The average MSE is 2.52 μm.

Figures (6)

• Figure 1: Concept overview of our coarse-to-fine learning approach for in vivo multi pipettes localisation under two-photon microscopy. The white cross "$\times$" in the left corner image indicates the position of the pipette tips.
• Figure 2: Framework Overview. The model takes the original in vivo image as input and first applies cycleGAN to enhance pipette tip features. The enhanced image, augmented with positional embeddings, is then processed by the encoder, which predicts a coarse heatmap of pipette tip locations. Finally, the decoder refines these predictions, generating precise pipette tip coordinates through the finer coordination learner.
• Figure 3: Workflow of cycleGAN-based pipette tip feature enhancement. A single pipette is first cropped from the in vivo two-photon microscopy images. CycleGAN is then applied to remove background noise, enhancing pipette tip visibility. Finally, the denoised single pipette images are mapped back to their original positions while preserving the original pipette tip coordinate labels.
• Figure 4: Physical setup of our robot-assisted in vivo two-photon targeted multi-patch-clamp system. RITA gonzalez2023two is the software that controls the robot and PMTs are the microscopy receptors of light.
• Figure 5: Visualisation of two datasets. (a) 3 examples from ex vivo Dataset gonzalez2021machine featuring a single pipette with a clear background. (b) 3 examples from our in-house in vivo dataset, captured using two-photon microscopy, containing multiple pipettes.