Learning Multimodal Volumetric Features for Large-Scale Neuron Tracing

TL;DR

To reduce the proofreading burden in EM-based neuron tracing, this work introduces FlyTracing, a dataset with $3.2\times 10^{6}$ positive segment-pair connections across the whole Drosophila brain, and a connectivity-aware contrastive learning framework to produce dense volumetric EM embeddings. The learned embeddings, optimized with a merge/split connectivity objective and a segmentation-clustering loss, are fused with 3D morphology representations (voxel masks or surface point clouds) to predict segment connectivity at scale, exhibiting robustness to artifacts such as misalignment and missing sections. Across large test blocks, the proposed Connect-Embed approach, especially when paired with PointNet++, outperforms baselines and improves tracing metrics (e.g., a $8.1\%$ relative gain in ERL for FAFB Mushroom Body). The dataset and accompanying code offer a scalable path toward automatic neuron tracing in connectomics and broader evaluation of multimodal connectivity strategies.

Abstract

The current neuron reconstruction pipeline for electron microscopy (EM) data usually includes automatic image segmentation followed by extensive human expert proofreading. In this work, we aim to reduce human workload by predicting connectivity between over-segmented neuron pieces, taking both microscopy image and 3D morphology features into account, similar to human proofreading workflow. To this end, we first construct a dataset, named FlyTracing, that contains millions of pairwise connections of segments expanding the whole fly brain, which is three orders of magnitude larger than existing datasets for neuron segment connection. To learn sophisticated biological imaging features from the connectivity annotations, we propose a novel connectivity-aware contrastive learning method to generate dense volumetric EM image embedding. The learned embeddings can be easily incorporated with any point or voxel-based morphological representations for automatic neuron tracing. Extensive comparisons of different combination schemes of image and morphological representation in identifying split errors across the whole fly brain demonstrate the superiority of the proposed approach, especially for the locations that contain severe imaging artifacts, such as section missing and misalignment. The dataset and code are available at https://github.com/Levishery/Flywire-Neuron-Tracing.

Figures (6)

• Figure 1: The pipeline of large-scale neuron reconstruction consists of EM image over-segmentation and human proofreading for complete neuron reconstruction. Each neuron is represented by a tree-structure skeleton.
• Figure 2: Example of sample pairs from FlyTracing. The first row shows their 3D meshes (segment instances denoted by color), with the representative EM image slices shown below. To determine whether two segments belong to the same neuron, human tracers frequently cross-reference their 3D meshes and adjacent EM image slices.
• Figure 3: Our connectivity prediction framework fuses local volumetric image features extracted by the EmbedNet with 3D morphology, optionally represented by point cloud or volumetric masks.
• Figure 4: Our volumetric image embedding network. For a pair of adjacent segments, a small EM volume centered at the truncation point is cropped and fed into EmbedNet to extract per-voxel $k$-dimensional embeddings. The EmbedNet is trained via a contrastive loss based on the pairwise segment connectivity of the query segment and the other neighboring segments.
• Figure 5: Precision-recall curves of different models on challenging blocks with image degradation. The triangle markers denote the performance using threshold $f(S_a,S_b)>0.5$.