Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Representation Learning in a Decomposed Encoder Design for Bio-inspired Hebbian Learning

Achref Jaziri, Sina Ditzel, Iuliia Pliushch, Visvanathan Ramesh

TL;DR

The paper tackles learning robust representations under biologically plausible, local Hebbian rules by introducing a decomposed encoder design with multiple invariant operators. It trains parallel encoders via a bio-inspired contrastive predictive coding framework (CLAPP or HingeCPC) and evaluates them with a downstream linear classifier, using datasets such as GTSRB, STL10, and CODEBRIM (with video experiments in the appendix). Key findings show that carefully chosen invariants (e.g., LBP, RG normalization, DTCWT) substantially improve robustness and close the gap to backpropagation, often outperforming non-decomposed baselines, and achieving competitive multi-target performance on CODEBRIM. The work highlights the importance of integrating domain knowledge and inductive biases with local learning rules to obtain transparent, generalizable representations suitable for real-world tasks.

Abstract

Modern data-driven machine learning system designs exploit inductive biases in architectural structure, invariance and equivariance requirements, task-specific loss functions, and computational optimization tools. Previous works have illustrated that human-specified quasi-invariant filters can serve as a powerful inductive bias in the early layers of the encoder, enhancing robustness and transparency in learned classifiers. This paper explores this further within the context of representation learning with bio-inspired Hebbian learning rules. We propose a modular framework trained with a bio-inspired variant of contrastive predictive coding, comprising parallel encoders that leverage different invariant visual descriptors as inductive biases. We evaluate the representation learning capacity of our system in classification scenarios using diverse image datasets (GTSRB, STL10, CODEBRIM) and video datasets (UCF101). Our findings indicate that this form of inductive bias significantly improves the robustness of learned representations and narrows the performance gap between models using local Hebbian plasticity rules and those using backpropagation, while also achieving superior performance compared to non-decomposed encoders.

Representation Learning in a Decomposed Encoder Design for Bio-inspired Hebbian Learning

TL;DR

The paper tackles learning robust representations under biologically plausible, local Hebbian rules by introducing a decomposed encoder design with multiple invariant operators. It trains parallel encoders via a bio-inspired contrastive predictive coding framework (CLAPP or HingeCPC) and evaluates them with a downstream linear classifier, using datasets such as GTSRB, STL10, and CODEBRIM (with video experiments in the appendix). Key findings show that carefully chosen invariants (e.g., LBP, RG normalization, DTCWT) substantially improve robustness and close the gap to backpropagation, often outperforming non-decomposed baselines, and achieving competitive multi-target performance on CODEBRIM. The work highlights the importance of integrating domain knowledge and inductive biases with local learning rules to obtain transparent, generalizable representations suitable for real-world tasks.

Abstract

Modern data-driven machine learning system designs exploit inductive biases in architectural structure, invariance and equivariance requirements, task-specific loss functions, and computational optimization tools. Previous works have illustrated that human-specified quasi-invariant filters can serve as a powerful inductive bias in the early layers of the encoder, enhancing robustness and transparency in learned classifiers. This paper explores this further within the context of representation learning with bio-inspired Hebbian learning rules. We propose a modular framework trained with a bio-inspired variant of contrastive predictive coding, comprising parallel encoders that leverage different invariant visual descriptors as inductive biases. We evaluate the representation learning capacity of our system in classification scenarios using diverse image datasets (GTSRB, STL10, CODEBRIM) and video datasets (UCF101). Our findings indicate that this form of inductive bias significantly improves the robustness of learned representations and narrows the performance gap between models using local Hebbian plasticity rules and those using backpropagation, while also achieving superior performance compared to non-decomposed encoders.
Paper Structure (12 sections, 3 equations, 2 figures, 3 tables)

This paper contains 12 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Decomposition
  5. Local Learning with Contrastive Predictive Coding
  6. Experiments
  7. Experimental Setting
  8. Experimental Results
  9. Ablations
  10. Multi-Target Classification
  11. Conclusion
  12. Acknowledgments

Figures (2)

  • Figure 1: An illustration of the presented framework. Each encoder network is preceded by a transformation and is trained in a contrastive learning setting. Afterwards, the linear classifier is trained on a downstream classification task while the weights of the encoders are frozen.
  • Figure 2: Qualitative results illustrating t-SNE dimensionality reduction of the latent encodings on GTSRB of CLAPP (left plot) and our framework trained locally (right plot) models. We visualize only 4 classes of street signs to avoid clutter. The following classes were randomly chosen: Speed Limit 20km/h sign (blue), Turn Straight Sign (green), No way general sign (orange ), Attention bottleneck sign (red). Further visualizations of t-SNE and PCA dimensionality reduction are included in the appendix.