Rethinking Generalizability and Discriminability of Self-Supervised Learning from Evolutionary Game Theory Perspective

TL;DR

This work reveals a mutual-exclusion between generalizability and discriminability in self-supervised representations and proposes a unified framework, ESSL, that leverages evolutionary game theory to guide a balanced trade-off. By decomposing SSL into a generalizability- and a discriminability-focused branch and optimizing their balance with reinforcement learning (PPO), ESSL tightens the theoretical generalization bound and achieves state-of-the-art performance on both conventional and large-scale benchmarks. Empirically, ESSL variants outperform baselines across multiple datasets, and visualizations indicate that the method simultaneously captures global semantic structure and local discriminative cues. The approach holds practical impact for deploying SSL in diverse domains by delivering robust, transferable representations without sacrificing discriminability.

Abstract

Representations learned by self-supervised approaches are generally considered to possess sufficient generalizability and discriminability. However, we disclose a nontrivial mutual-exclusion relationship between these critical representation properties through an exploratory demonstration on self-supervised learning. State-of-the-art self-supervised methods tend to enhance either generalizability or discriminability but not both simultaneously. Thus, learning representations jointly possessing strong generalizability and discriminability presents a specific challenge for self-supervised learning. To this end, we revisit the learning paradigm of self-supervised learning from the perspective of evolutionary game theory (EGT) and outline the theoretical roadmap to achieve a desired trade-off between these representation properties. EGT performs well in analyzing the trade-off point in a two-player game by utilizing dynamic system modeling. However, the EGT analysis requires sufficient annotated data, which contradicts the principle of self-supervised learning, i.e., the EGT analysis cannot be conducted without the annotations of the specific target domain for self-supervised learning. Thus, to enhance the methodological generalization, we propose a novel self-supervised learning method that leverages advancements in reinforcement learning to jointly benefit from the general guidance of EGT and sequentially optimize the model to chase the consistent improvement of generalizability and discriminability for specific target domains during pre-training. Theoretically, we establish that the proposed method tightens the generalization error upper bound of self-supervised learning. Empirically, our method achieves state-of-the-art performance on various benchmarks.

Figures (10)

• Figure 1: Generalizability and discriminability among the generalizability model (GEN), the discriminability model (DIS), ESSL w/ Ensemble, and ESSL. The generalizability model, implemented by SimCLR$^\dagger$ which is a variant of SimCLR, demonstrates better generalizability, while the discriminability model, implemented by Barlow Twins, exhibits superior discriminability. ESSL w/ Ensemble, a straightforward combination of the generalizability model and the discriminability model, results in a reduction of both generalizability and discriminability. In contrast, our proposed ESSL method effectively maintains both generalizability and discriminability simultaneously. The values of generalizability and discriminability are computed using Equation \ref{['equ:gen_dis']}, and the details of the four methods mentioned above can be found in Appendix \ref{['app:fig1detail']}.
• Figure 2: The framework of ESSL. The module in green denotes the training procedure of the SSL model. The module in blue is the modeled Markov Decision Processes of RL to adjust weights for the generalizability loss, i.e., $\mathcal{L}_{GEN}$, and the discriminability loss, i.e., $\mathcal{L}_{DIS}$, in a self-paced manner.
• Figure 3: Phase diagram for the dynamic system. $x$ and $y$ represent the proportion of discriminability and generalizability models, respectively. Every line denotes the evolving strategy of the candidate models under different initialization.
• Figure 4: Decision tree illustration for the EGT model between generalizability and discriminability.
• Figure 5: The proposed SCM for modeling the causality encompassing the generalizability and discriminability of SSL from the perspective of data generalization, which is illustrated in (a). (b) and (c) are the SCMs highlighting the discriminability causal path and the generalizability causal path, respectively.

Theorems & Definitions (4)

• theorem 1
• lemma thmcounterlemma
• corollary thmcountercorollary
• proof