Hierarchical temporal receptive windows and zero-shot timescale generalization in biologically constrained scale-invariant deep networks

TL;DR

The paper investigates how cortical hierarchies of temporal processing can emerge despite heterogeneous local time constants. It first shows that a scale-invariant feedforward model (SITHCon) develops a hierarchy of Temporal Receptive Windows (TRWs) across layers when trained on a hierarchical toy language. It then derives a family of recurrent architectures (SITH-RNN) with inductive priors—block-diagonal structure, geometric time constants, and translation-invariant readouts—that yield scale-invariant, dual-timecell dynamics and perfect zero-shot generalization to time-scaled inputs with far fewer parameters. The findings suggest that brain-like narrative processing can be achieved through specific architectural priors, supporting a normative view of scale-invariant temporal memory and its relevance for robust sequence modeling in AI. Overall, the work links biological time cells and temporal context cells to computational architectures that efficiently encode and generalize across multiple timescales, offering insights for both neuroscience and scalable AI systems.

Abstract

Human cognition integrates information across nested timescales. While the cortex exhibits hierarchical Temporal Receptive Windows (TRWs), local circuits often display heterogeneous time constants. To reconcile this, we trained biologically constrained deep networks, based on scale-invariant hippocampal time cells, on a language classification task mimicking the hierarchical structure of language (e.g., 'letters' forming 'words'). First, using a feedforward model (SITHCon), we found that a hierarchy of TRWs emerged naturally across layers, despite the network having an identical spectrum of time constants within layers. We then distilled these inductive priors into a biologically plausible recurrent architecture, SITH-RNN. Training a sequence of architectures ranging from generic RNNs to this restricted subset showed that the scale-invariant SITH-RNN learned faster with orders-of-magnitude fewer parameters, and generalized zero-shot to out-of-distribution timescales. These results suggest the brain employs scale-invariant, sequential priors - coding "what" happened "when" - making recurrent networks with such priors particularly well-suited to describe human cognition.

Figures (11)

• Figure 1: There is evidence to suggest that brain regions, each with a heterogeneity of time-scales, can still exhibit a hierarchy of increasing Temporal Receptive Windows.Left: Spoken language has a multitude of timescales, where one has to keep simultaneously keep track of words, sentences and the paragraphs to make sense of the current word (a1). Different brain regions need different timescales of context to process the current input, defined as its Temporal Receptive Window (TRW). To measure this processing timescale, there has been a body of work which measure the correlation of activity from a brain region in response to naturalistic stimuli, both intact and scrambled (a2). Scrambling the sequence at different timescales corresponds to swapping out different lengths of context preceding an input (a3), and tests how sensitive a brain region is to these different scales of context (b). These studies widely support the hypothesis that there is a hierarchy of temporal receptive windows in the cortex (c), with sensory areas processing the immediate context preceding the input, while higher-order areas collect and assimilate context from longer timescales like sentences and paragraphs. Right: Recent studies in memory neuroscience show populations of cells called temporal context cells (e.) and time cells (f.) seen primarily in the hippocampus and some other brain regions, both of which encode a compressed record of the recent past, using different temporal receptive fields. These populations show a continuous distribution of single-neuron time constants with more cells coding for the recent past. Together with other converging evidence about a heterogeneity of single-neuron timescales within brain regions, these raise the question of whether the brain actually has a distribution of time constants within each brain region to support flexibility and rescaling---however, when processing sequences with a hierarchical temporal structure, these brain regions show different emerging processing timescales. Figure \ref{['fig:TRWschem']}a-c adapted from ChieHone20.
• Figure 2: A sequence of RNNs with inductive priors.a. Generic RNNs in neuroscience have feed-forward weights that project the input $x_t$ onto the hidden layer $(\mathbf{I})$, recurrent weights that operate on the hidden layer $(\mathbf{R})$, and feed-forward weights $(\mathbf{L})$ projecting the last hidden layer to the output (as shown in c1.). b.SITHCon is a feedforward convolutional neural network with inductive priors inspired from populations of time cells in the brain, which maintain a log-compressed record of the past. These priors grant it features like temporal scale-invariance and enable it to generalize to slower or faster input without retraining. Each layer in SITHCon has a matrix memory, with time cells with different time constants (When) keeping track of every feature (What). At the end of each layer, a convolution catches patterns in this log-compressed memory and creates features which can be tracked by the next layer. c. We propose a series of inductive priors that successively and gradually builds onto generic RNNs (c1) used in Neuroscience, building up a recurrent network very similar to SITHCon in c5. To emulate a What x When memory structure similar to SITHCon, we constrain the projection $(\mathbf{I})$ and recurrence matrices $(\mathbf{R})$ to have a block diagonal structure (c2). The What x When matrix memory thus appears here as a long hidden state, with each feature being tracked and evolved independently in its own temporal subspace. Similar to SITHCon, a set of output weights $(\mathbf{L})$ generate the output state, and a convolution + maxpool layer combines and remaps patterns from all features into new features for the next layer. We see that additional priors are required to gain scale-invariant and sequential activations, like diagonal Recurrence Matrices (introduced in c3), geometric eigenvalues in $(\mathbf{R})$ (introduced in c4), and translated motifs in $(\mathbf{L})$ (introduced in SITH-RNN, c5).
• Figure 3: A synthetic hierarchical language with structure across scales.a. Natural Language has a hierarchy of symbols, with each level combining to create the next. We need to keep track of information at these timescales simultaneously to make sense of the present. b. The toy language is constructed to have $9$ symbols in each level, with combinations of 3 'letters' forming a 'word', and so on. Only certain combinations of symbols are used, with different rules for each of the three positions---so that knowledge of the first two symbols is not enough to predict the third symbol. For example, the first 'letter' in each 'word' can only be the first three letters ($1,2,$ and $3$), the second positions are permutations of the middle letters ($4,5,$ and $6$), and the third position, and the third position can only be one of the last three 'letters' ($7,8,$ and $9$), with different rules to populate all possible words. The same motifs are used to generate the symbols from one level to the next---at each level, the symbol transitions are deterministic. c. The resultant toy language also has a hierarchy of levels, akin to natural language. The symbol transitions at lower levels will still change abruptly when boundaries of higher-level symbols are reached, and a network attempting to classify such a hierarchical sequence would thus need to keep track of symbols at multiple time-scales, akin to maintaining a recollection of the words, paragraphs, and larger context required to understand speech.
• Figure 4: Emergence of hierarchical temporal receptive windows in SITHCon.a. Recurrence plots reveal increasing temporal structure across layers. Plots show the temporal self-similarity of population vectors for the input (left) through Layer 4 (right). Rows show successive $3\times$ temporal magnifications (top to bottom). Early layers exhibit fine-grained temporal dynamics, while deeper layers maintain stable states over longer durations. b. Sensitivity to hierarchical scrambling. Akin to the methodology in TRW literature, we measured the correlation between responses to original sequences and sequences scrambled at varying timescales (schematic, top right). Early layers are sensitive only to fine-scale scrambling (small $s$) and remain robust to large-scale scrambling (high correlation at large $s$). In contrast, deep layers (e.g., Layer 4) depend on long-range structure and are therefore sensitive to scrambling at both fine and coarse scales (up to $3^4$), recovering only when the scrambling scale exceeds the length of the training sequence ($>3^4$). The grey dashed line indicates the 50% correlation threshold used to calculate the TRW. c. An emergent hierarchy of temporal integration windows. The effective Temporal Receptive Window (TRW) for each layer, calculated as the half-max scrambling scale ($s_{50}$) from the data in b using linear interpolation. The approximately linear trend on the semi-log plot indicates exponential expansion of integration timescales; the slight saturation at Layer 4 reflects the network matching the bounded global timescale of the training sequences ($>3^4$). d. Layer-wise tuning to hierarchical levels. Spike-Triggered Averages (STA) map linear receptive fields of representative neurons (columns) to stimuli at different hierarchical levels (rows). Layer 1 neurons (left) track elementary symbols but ignore higher-order structure. On the other hand, Layer 4 neurons (right) exhibit a nested compositional structure: a single broad activation band at the 'paragraph' level resolves into three distinct bands at the 'sentence' level, and so on, exhibiting signs that deep neurons have learned the hierarchical mapping of the toy language, defining their receptive fields through the specific combinatorial syntax of the grammar (for the RFs of all 9 neurons in each layer, refer to Fig. \ref{['fig:STAsupp']}).
• Figure 5: Biologically constrained SITH-RNNs achieve superior generalization with fewer parameters by enforcing scale-invariant dynamics.a. Evolution of architectural constraints. We derived a continuum of five recurrent networks, starting from a generic linear RNN (Left) and systematically adding inductive priors motivated by the SITHCon architecture (e.g., block-diagonal connectivity, geometric time constants), culminating in SITH-RNN (Right). b. Parameter efficiency. The addition of these structural constraints dramatically reduces the number of trainable weights (by orders of magnitude) compared to the generic RNN, despite identical hidden state dimensions. c. Zero-shot generalization to time-rescaling. When trained on hierarchical sequences at a fixed timescale ($3^0$) and tested on sequences rescaled by factors up to $3^6$, models perform increasingly well as priors are added (Left to Right). SITH-RNN achieves perfect classification accuracy across all six orders of magnitude of timescale, demonstrating robust zero-shot generalization. d. Spectral structure of recurrence. The eigenvalues of the recurrent matrix $\mathbf{R}$ transition from a uniform distribution on the complex unit circle (Generic RNN, Left) to real values localized on the axes (Block-Diagonal) and finally to a geometric spacing that tiles the real axis (SITH-RNN, Right), a necessary condition for logarithmic (Weber-Fechner) compression and scale-invariance. e. Temporal dynamics of the hidden state ($\mathbf{h}_t$). Response of the first-layer hidden state to a delta spike input at $t=0$. In networks with diagonal $\mathbf{R}$ (Right), neurons act as leaky integrators that decay smoothly at different rates, mirroring biological temporal context cells. f. Temporal dynamics of the readout ($\mathbf{Lh}_t$). Response of the projected output neurons. The banded matrix structure in $\mathbf{L}$ (specific to SITH-RNN, Right) generates translated eigenvectors, producing smooth, sequential activations with a spectrum of time constants that resemble biological time cells.