Better, But Not Sufficient: Testing Video ANNs Against Macaque IT Dynamics

TL;DR

This study tests whether video-trained artificial neural networks can account for the dynamic temporal responses of macaque IT during naturalistic viewing. By recording IT activity and benchmarking against static, recurrent, and video-based models, the authors show modest late-phase gains for video models but reveal a crucial limitation: appearance-free motion in IT generalizes beyond appearance cues, which current video models fail to capture. The stress-test decoders demonstrate that IT encodes appearance-invariant temporal statistics that are not reproduced by existing architectures or training schemes. The results argue for new objectives and architectural biases that encode biological temporal invariances, moving beyond time-unfolded feedforward or shallow recurrence toward truly dynamic Vision models.

Abstract

Feedforward artificial neural networks (ANNs) trained on static images remain the dominant models of the the primate ventral visual stream, yet they are intrinsically limited to static computations. The primate world is dynamic, and the macaque ventral visual pathways, specifically the inferior temporal (IT) cortex not only supports object recognition but also encodes object motion velocity during naturalistic video viewing. Does IT's temporal responses reflect nothing more than time-unfolded feedforward transformations, framewise features with shallow temporal pooling, or do they embody richer dynamic computations? We tested this by comparing macaque IT responses during naturalistic videos against static, recurrent, and video-based ANN models. Video models provided modest improvements in neural predictivity, particularly at later response stages, raising the question of what kind of dynamics they capture. To probe this, we applied a stress test: decoders trained on naturalistic videos were evaluated on "appearance-free" variants that preserve motion but remove shape and texture. IT population activity generalized across this manipulation, but all ANN classes failed. Thus, current video models better capture appearance-bound dynamics rather than the appearance-invariant temporal computations expressed in IT, underscoring the need for new objectives that encode biological temporal statistics and invariances.

Figures (2)

• Figure 1: A. Neural predictivity of early and late features extracted from early and late video frames by static feed-forward and recurrent ANNs. We report mean and standard error across neurons. B. (Left) Neural predictivity of early and late static and video ANNs on early neural responses. We report mean and standard error across neurons. (Right) Per-neuron explained variance by the best static and video model in early responses. We report mean and standard error across repetitions. C. (Left) Neural predictivity of early and late static and video ANNs on late neural responses. We report mean and standard error across neurons. (Right) Per-neuron explained variance by the best static and video model in late responses. We report mean and standard error across repetitions.
• Figure 2: A. LSTM-based decoding mechanism used to evaluate how well the model represents dynamic object properties in natural videos. B. Decoding accuracy of object-motion direction from late model features and late IT responses (83 neurons) for static and video ANNs on natural videos. We report mean and standard error across videos. C. LSTM-based decoding mechanism used to evaluate how well the model represents dynamic object properties in appearance-free videos. Appearance-free videos consist of frame sequences where appearance is replaced by random pixel values that move according to the original object motion ilic2022appearance. D. Decoding accuracy of object-motion direction from late model features and late IT responses (83 neurons) for static and video ANNs on appearance-free videos. We report mean and standard error across videos.