Machine Learning Modeling for Multi-order Human Visual Motion Processing

TL;DR

This work introduces a human-aligned motion perception model that combines trainable motion-energy sensing with a recurrent motion-graph integration in a two-stage V1-MT-inspired architecture. A complementary higher-order pathway, realized with a multilayer 3D-CNN, enables robust second-order motion processing, particularly when trained on non-Lambertian materials. The model achieves dense optical-flow performance competitive with state-of-the-art CV methods while reproducing human psychophysics and neurophysiological findings, including segmentation via motion-based graph cuts without extra training. By linking material properties, second-order motion, and global motion integration, the approach provides a principled framework for robust motion understanding in natural scenes and offers insights into biological motion processing.

Abstract

Our research aims to develop machines that learn to perceive visual motion as do humans. While recent advances in computer vision (CV) have enabled DNN-based models to accurately estimate optical flow in naturalistic images, a significant disparity remains between CV models and the biological visual system in both architecture and behavior. This disparity includes humans' ability to perceive the motion of higher-order image features (second-order motion), which many CV models fail to capture because of their reliance on the intensity conservation law. Our model architecture mimics the cortical V1-MT motion processing pathway, utilizing a trainable motion energy sensor bank and a recurrent graph network. Supervised learning employing diverse naturalistic videos allows the model to replicate psychophysical and physiological findings about first-order (luminance-based) motion perception. For second-order motion, inspired by neuroscientific findings, the model includes an additional sensing pathway with nonlinear preprocessing before motion energy sensing, implemented using a simple multilayer 3D CNN block. When exploring how the brain acquired the ability to perceive second-order motion in natural environments, in which pure second-order signals are rare, we hypothesized that second-order mechanisms were critical when estimating robust object motion amidst optical fluctuations, such as highlights on glossy surfaces. We trained our dual-pathway model on novel motion datasets with varying material properties of moving objects. We found that training to estimate object motion from non-Lambertian materials naturally endowed the model with the capacity to perceive second-order motion, as can humans. The resulting model effectively aligns with biological systems while generalizing to both first- and second-order motion phenomena in natural scenes.

Figures (6)

• Figure 1: The Two-Stage Motion Perception System Prototype(A): The first stage (Stage-I) mimics V1 function by detecting local motion. The second stage (Stage II) leverages a graph network for recurrent motion integration and segregation. Stage I incorporates dual channels to process both first-order and higher-order motion. The first-order channel captures Fourier-based motion in motion energy units, thus via the red route. The higher-order channel employs normative 3D CNN layers to extract higher-order features via the gray route. Natural videos are used to train the entire model to estimate motion flow. (B): Illustration of spatiotemporal receptive fields of motion energy units in Stage I, resembling those of V1 neurons, which are sensitive to drifting gratings with specific spatiotemporal frequencies and directions of movements.
• Figure 2: Recurrent Motion Integration and Evaluation in Experimental Setting.(A): Direction tuning across the neuronal groups when both 1D and 2D stimuli were imparted. The animal data are redrawn from movshon1992analysis. (B): Motion integration connects local movements to solve the aperture problem and to globally interpret motion. This aligns with human perception of adaptive pooling amano2009adaptive. (C) Motion integration is sensitive to higher-order pattern cues. We used the three scenarios A, B, and C detailed in mcdermott2001beyond. The extents of integration were quantified by correlating the directions of motion between adjacent segments across a single rotation cycle. As compared to scenario C, scenario B, characterized by structural constraints and depth cues, led to an increased integration index in the model, similar to human perception mcdermott2001beyond. In the middle column of the right panels (for unit connections), we visualize the attention heat map derived from the motion graph, showing the connectivity of the unit (marked by a circle) with other units in Stage II.
• Figure 3: Recurrent Motion Integration and Evaluation in Natural Contexts(A): Recurrent integration in natural scenarios. "Stage II-$N$" refers to the outcomes from the $N$th iteration in Stage II, with visualizations illustrating neuron connectivity via heat maps. The neural connectivity is represented as a graph structure, and through graph bipartitioning shi2000normalized, the model can further achieve instance segmentation without any additional training. (B, C): A comparison of the motion estimations of our model and human perception using the Sintel Benchmark. In (B), the shadow around the red line represents the 95% confidence interval (CI) of the linear regression line. In (C), larger red circles show a closer response to humans over the GT, while blue circles indicate the reverse. Despite being trained to fit the GT, our model uniquely exhibits a higher correlation with human responses than with the GT.
• Figure 4: A Material-Controlled Motion Dataset and a Second-order Benchmark Demonstration(A): We manipulated the material properties to create two motion datasets with optical flow labels. One featured purely reflective materials. The other included non-Lambertian materials such as specular, glossy, translucent, and anisotropic surfaces. (B): A large second-order benchmark was generated by applying naturalistic modulations, thus water waves and swirl effects, to natural images. (C): The psychophysical experiments performed using these datasets revealed that humans reliably and accurately perceive a range of second-order motions. Machine vision models cannot yet do this. The accompanying figure illustrates the perceived motion vectors from a single participant's data. The shadow around the fitted line represents the 95% CI
• Figure 5: The Interplay Between Material Properties and Second-order Motion Perception(A) We computed the average Pearson correlations across various types of second-order motion and contrasted these to those of contemporary CV models. Our dual-channel model, trained on a non-diffuse dataset, exhibited superior performance. The proficiency was near-human. The error bars show the 95% CI across seven modulation types; Ptcp-N represent different participants. (B): The directional tuning curves of model units were tested using both first- and second-order gratings. A modified circular variance measure was employed to quantify directional tuning. This ranged from 0 to 1, where 0 indicates a uniform distribution. We present the results for units in the first- and higher-order channels trained on both the diffuse and non-diffuse datasets. (C): Detailed Pearson Correlations with the Human Responses across all Modulation Types. Mod1 to Mod7 are various second-order modulations, including random noise, Gaussian blur, water wave, Fourier phase shuffle, random pixel shuffle, swirl, and drift-balanced motion. The dual-channel model trained on non-diffuse datasets exhibited significantly improved recognition of second-order motion. The error bars show the 95% CI within each modulation type.