Coupled Transformer Autoencoder for Disentangling Multi-Region Neural Latent Dynamics

TL;DR

This work introduces the Coupled Transformer Autoencoder (CTAE), a Transformer-based framework that disentangles shared and region-specific latent dynamics from multi-region neural recordings while modeling non-stationary nonlinear temporal structure. By partitioning each region’s latent space into orthogonal shared and private components and enforcing alignment and orthogonality through targeted losses, CTAE yields interpretable cross-region interactions and improves behaviorally relevant decoding without retraining downstream readouts. Empirical validation on two datasets (M1–PMd and SC–ALM) shows CTAE captures more shared variance and reveals anatomically consistent inter-regional couplings, outperforming prior multi-region latent-variable models. The approach scales to more regions and general multiview time series, offering a principled, flexible tool for studying distributed neural computations and guiding causal interventions.

Abstract

Simultaneous recordings from thousands of neurons across multiple brain areas reveal rich mixtures of activity that are shared between regions and dynamics that are unique to each region. Existing alignment or multi-view methods neglect temporal structure, whereas dynamical latent variable models capture temporal dependencies but are usually restricted to a single area, assume linear read-outs, or conflate shared and private signals. We introduce the Coupled Transformer Autoencoder (CTAE) - a sequence model that addresses both (i) non-stationary, non-linear dynamics and (ii) separation of shared versus region-specific structure in a single framework. CTAE employs transformer encoders and decoders to capture long-range neural dynamics and explicitly partitions each region's latent space into orthogonal shared and private subspaces. We demonstrate the effectiveness of CTAE on two high-density electrophysiology datasets with simultaneous recordings from multiple regions, one from motor cortical areas and the other from sensory areas. CTAE extracts meaningful representations that better decode behavioral variables compared to existing approaches.

Figures (16)

• Figure 1: Observed neural activity across time from two brain regions, denoted as $\bm{X}^{(1)}$ and $\bm{X}^{(2)}$, is modeled as a nonlinear function of underlying latent dynamics specific to each region. In the illustration, $[e_1, e_2, e_3]$ span the latent subspace for region 1 and $[e_1, e_2, e_4]$ for region 2. Inter-regional communication is mediated by shared latent trajectories $S$ within the common subspace $[e_1, e_2]$, which drive correlated population activity. Following the output-null/potent hypothesis, we assume shared and private dimensions are orthogonal, allowing clear recovery of shared dynamics from region-specific processes.
• Figure 2: CTAE architecture.CTAE is composed of a coupled autoencoder, where the encoders and decoders are causal transformers designed to reconstruct neural activity for each region $r$. The inputs to the network are estimated spike rates from each region. A weight mask per region $w^{(r)}$ is used to disentangle the shared representation (violet) from the region-specific latents (red and blue) in the encoder outputs $Z^{(r)}$. The latents are recovered via end-to-end training.
• Figure 3: M1-PMd dataset. CTAE shared (top) and region-private (bottom) latents. Dotted vertical line indicates the “go” cue in each trial.
• Figure 4: M1-PMd dataset. a) Ground truth hand position in top-left corner. Hand position decoding from neural activity in M1 and PMd (top), from CTAE latents (middle) and from DLAG (bottom). b) Confusion matrices for reach direction classification (order of plots is same as in A). Classification accuracy in parentheses.
• Figure 5: SC-ALM dataset. Representative latents. Each panel shows the condition-averaged time course of one latent (title = subspace and latent index). Left panels: stimuli—visual (blue), tactile (orange), multisensory (green). Right panels: target side—right (red), left (purple).