Improving Unsupervised Task-driven Models of Ventral Visual Stream via Relative Position Predictivity
Dazhong Rong, Hao Dong, Xing Gao, Jiyu Wei, Di Hong, Yaoyao Hao, Qinming He, Yueming Wang
TL;DR
The paper addresses the limitation of contrastive unsupervised task-driven models of the ventral visual stream by incorporating Relative Position Learning (RPL) to capture RP prediction in addition to object recognition. It introduces a dual-task framework that combines SimCLR-style contrastive loss $\mathcal{L}_{\text{CL}}$ with an RP loss $\mathcal{L}_{\text{RPL}}$, optimized as $\mathcal{L} = \mathcal{L}_{\text{CL}} + \alpha \mathcal{L}_{\text{RPL}}$, using a ResNet-18 base and simple MLP heads. Experimental results on STL-10 and macaque neural data show that increasing $\alpha$ improves RP predictivity and often boosts image recognition accuracy, while RP predictivity significantly enhances brain similarity across V1, V2, V4, and IT, with a layer-wise pattern indicating shallow layers align with V1 and deeper layers with IT. These findings provide computational evidence that VVS participates in RP prediction, suggesting more biologically realistic unsupervised representations can be learned by jointly modeling object recognition and spatial localization. The work advances brain-aligned modeling of the VVS and lays groundwork for future biologically plausible architectures and richer cortical mappings.
Abstract
Based on the concept that ventral visual stream (VVS) mainly functions for object recognition, current unsupervised task-driven methods model VVS by contrastive learning, and have achieved good brain similarity. However, we believe functions of VVS extend beyond just object recognition. In this paper, we introduce an additional function involving VVS, named relative position (RP) prediction. We first theoretically explain contrastive learning may be unable to yield the model capability of RP prediction. Motivated by this, we subsequently integrate RP learning with contrastive learning, and propose a new unsupervised task-driven method to model VVS, which is more inline with biological reality. We conduct extensive experiments, demonstrating that: (i) our method significantly improves downstream performance of object recognition while enhancing RP predictivity; (ii) RP predictivity generally improves the model brain similarity. Our results provide strong evidence for the involvement of VVS in location perception (especially RP prediction) from a computational perspective.