Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PhiNet v2: A Mask-Free Brain-Inspired Vision Foundation Model from Video

Makoto Yamada, Kian Ming A. Chai, Ayoub Rhim, Satoki Ishikawa, Mohammad Sabokrou, Yao-Hung Hubert Tsai

TL;DR

PhiNet v2 advances brain-inspired vision foundation modeling by deploying a Transformer-based encoder to learn robust video representations from sequential input without strong data augmentation. It grounds learning in a variational framework with latent variables and two hippocampal/neocortical losses (Sim-1 and Sim-2), aligned with CLS theory and predictive coding. The paper presents detailed probabilistic derivations, architectural choices (ViT encoder, linear CA3 predictor, cross-attentive CA1 predictor), and extensive ablations showing the importance of EMA, symmetric loss, and careful learning dynamics. Empirically, PhiNet v2 delivers competitive performance against state-of-the-art video SSL methods (e.g., RSP, CropMAE) across DAVIS, VIP, and JHMDB, while avoiding MAE-style auxiliary modules and enabling learning from continuous streams with practical training settings.

Abstract

Recent advances in self-supervised learning (SSL) have revolutionized computer vision through innovative architectures and learning objectives, yet they have not fully leveraged insights from biological visual processing systems. Recently, a brain-inspired SSL model named PhiNet was proposed; it is based on a ResNet backbone and operates on static image inputs with strong augmentation. In this paper, we introduce PhiNet v2, a novel Transformer-based architecture that processes temporal visual input (that is, sequences of images) without relying on strong augmentation. Our model leverages variational inference to learn robust visual representations from continuous input streams, similar to human visual processing. Through extensive experimentation, we demonstrate that PhiNet v2 achieves competitive performance compared to state-of-the-art vision foundation models, while maintaining the ability to learn from sequential input without strong data augmentation. This work represents a significant step toward more biologically plausible computer vision systems that process visual information in a manner more closely aligned with human cognitive processes.

PhiNet v2: A Mask-Free Brain-Inspired Vision Foundation Model from Video

TL;DR

PhiNet v2 advances brain-inspired vision foundation modeling by deploying a Transformer-based encoder to learn robust video representations from sequential input without strong data augmentation. It grounds learning in a variational framework with latent variables and two hippocampal/neocortical losses (Sim-1 and Sim-2), aligned with CLS theory and predictive coding. The paper presents detailed probabilistic derivations, architectural choices (ViT encoder, linear CA3 predictor, cross-attentive CA1 predictor), and extensive ablations showing the importance of EMA, symmetric loss, and careful learning dynamics. Empirically, PhiNet v2 delivers competitive performance against state-of-the-art video SSL methods (e.g., RSP, CropMAE) across DAVIS, VIP, and JHMDB, while avoiding MAE-style auxiliary modules and enabling learning from continuous streams with practical training settings.

Abstract

Recent advances in self-supervised learning (SSL) have revolutionized computer vision through innovative architectures and learning objectives, yet they have not fully leveraged insights from biological visual processing systems. Recently, a brain-inspired SSL model named PhiNet was proposed; it is based on a ResNet backbone and operates on static image inputs with strong augmentation. In this paper, we introduce PhiNet v2, a novel Transformer-based architecture that processes temporal visual input (that is, sequences of images) without relying on strong augmentation. Our model leverages variational inference to learn robust visual representations from continuous input streams, similar to human visual processing. Through extensive experimentation, we demonstrate that PhiNet v2 achieves competitive performance compared to state-of-the-art vision foundation models, while maintaining the ability to learn from sequential input without strong data augmentation. This work represents a significant step toward more biologically plausible computer vision systems that process visual information in a manner more closely aligned with human cognitive processes.
Paper Structure (24 sections, 9 equations, 3 figures, 9 tables, 1 algorithm)

This paper contains 24 sections, 9 equations, 3 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. PhiNets
  4. Brain-Inspired Interpretation
  5. Probabilistic derivation
  6. Considerations for learning
  7. Overall Objective
  8. Implementation of PhiNet v2
  9. Experiments
  10. Results
  11. Ablation Study
  12. PhiNet v1 with Transformer encoder (22.2)
  13. Exponential Moving Average (34.9)
  14. StopGradients (55.8, 59.7)
  15. Symmetric Loss (58.0, 58.7)
  16. ...and 9 more sections

Figures (3)

  • Figure 1: The architectures of PhiNet v1 (X-PhiNet in ishikawa2025phinets), RSP jang2024visual, and PhiNet v2 are shown, illustrating that PhiNet v2 combines PhiNet v1 and RSP. EMA stands for exponential moving average; SG-1, SG-2, and SG-prior refer to the stop-gradient mechanism; and Sim-1 and Sim-2 are similarity/discrepancy functions. The MAE module of RSP, which improves its performance, not shown, and it is not required by PhiNet v2.
  • Figure 2: Qualitative results on the DAVIS dataset. Top: Parkour. Bottom: Horsejump. "Ref" denotes the ground-truth mask in the first frame. "25%" and "75%" correspond to intermediate predictions, while "100%" shows the prediction on the final frame.
  • Figure 3: The biological circuit interpretation of PhiNet v1 (X-PhiNet in ishikawa2025phinets) and PhiNet v2.