Asynchronous-spectral fusion fluorescence microscopy for microsecond-scale behavioral dynamics

Abstract

Event-based image sensors provide microsecond temporal resolution but lack spectral discrimination, whereas diffractive spectral imagers encode wavelength information at conventional frame rates. We introduce a fluorescence microscopy architecture that fuses asynchronous event streams with diffraction-encoded CMOS measurements to decouple temporal and spectral sampling. The system achieves ~3.9 um spatial resolution over a 0.5 mm field of view, effective temporal resolution down to 100 us, and differentiates fluorophores whose emission peaks are separated by only 23 nm. By synchronizing and computationally merging both sensing modalities, we enable spectrally resolved tracking at 100,000 frames/s without scanning or filter switching.

Figures (4)

• Figure 1: Sensor-fusion fluorescence microscope. (a) A 50/50 beam splitter directs fluorescence simultaneously to a CMOS image sensor (CIS) and a dynamic vision sensor (DVS). A diffractive optical element (DOE) in the CIS arm encodes wavelength-dependent structure into a diffractogram relayed by a 1:1 4F system, while the DVS records asynchronous events from the same field of view. (b) Diffractogram of a USAF target demonstrating $\sim$2.2 $\mu$m diffraction-limited resolution. (c) A neural network performs pixel-level classification between spectrally overlapping fluorophores (mKate2 and mCherry; emission-peak separation = 23 nm, $>$62% overlap).
• Figure 2: Spectral classification and ultra-fast tracking. (a) Representative diffractogram from a static specimen. (b) Pixel-level neural-network classification separating mKate2 and mCherry into a color-coded map. (c) Confusion matrix for pixel classification (mKate2, mCherry, background), showing 96% average accuracy. (d) Spectrally resolved regions of a single C. elegans tracked at sub-millisecond temporal resolution after fusion with DVS event data. Inset: worm schematic indicating mKate2 (green) and mCherry (red) labeling.
• Figure 3: Non-rigid motion tracking. (a) Overview of the event-based motion-tracking pipeline. Asynchronous DVS events are clustered, registered using Gaussian Mixture Model (GMM) point-set alignment, and temporally filtered with a Kalman estimator to produce continuous trajectories. (b) Representative tracking of a freely moving C. elegans within the field of view over $\sim$10 s.
• Figure 4: Blue-light–evoked acceleration. Blue-light stimulation response of wild-type (WT) and mutant (UNC) worms in top and bottom panels, respectively. Events are accumulated over 10 ms temporal windows centered around the onset of blue excitation (dashed line indicates t = 0, when blue light turns on). The lower temporal resolution was chosen to match the fastest observed worm dynamics. Orange (positive-polarity) events mark pixels where intensity increased as the worm moved away, whereas blue (negative-polarity) events indicate intensity decreases due to occlusion as the worm moved toward those locations. When the two polarities remain spatially close (pre-stimulus), displacement within the accumulation window is minimal. Following blue-light onset, the separation between polarities increases, reflecting rapid acceleration and coiling behavior. Wild-type animals initiate this response within $\sim$0.18 s, whereas unc-101 mutants exhibit delayed coiling with latency approaching 0.88 s. Also see Supplementary videos 1 and 2.