Exploring Transferability of Self-Supervised Learning by Task Conflict Calibration

TL;DR

This work addresses the transferability of self-supervised representations and identifies task conflict as a key barrier when SSL is trained with multiple tasks per batch. It introduces Task Conflict Calibration (TC^2), which constructs multiple SSL tasks within each batch and learns a shared factor basis $V$ and per-sample weights $W_j=f_w(z_j)V^T$ to isolate task-relevant semantics via a two-stage bi-level optimization. The method enforces factor orthogonality and data reconstruction through $\mathcal{L}_v$ and promotes sparsity in sample-factor weights via $\mathcal{L}_w$, combining them into $\mathcal{L}_{tc^2}=\mathcal{L}_v+\lambda_s\mathcal{L}_w$; TC^2 is then optimized in a two-stage process to maximize transferability while suppressing inter-task interference. Empirical results across transfer learning, video-based tasks, and OOD benchmarks show consistent improvements over strong SSL baselines, validating the approach and underscoring the importance of modeling task distributions within SSL pipelines. Overall, TC^2 provides a practical, plug-in mechanism to improve SSL transferability by calibrating task-level representations and mitigating task conflicts.

Abstract

In this paper, we explore the transferability of SSL by addressing two central questions: (i) what is the representation transferability of SSL, and (ii) how can we effectively model this transferability? Transferability is defined as the ability of a representation learned from one task to support the objective of another. Inspired by the meta-learning paradigm, we construct multiple SSL tasks within each training batch to support explicitly modeling transferability. Based on empirical evidence and causal analysis, we find that although introducing task-level information improves transferability, it is still hindered by task conflict. To address this issue, we propose a Task Conflict Calibration (TC$^2$) method to alleviate the impact of task conflict. Specifically, it first splits batches to create multiple SSL tasks, infusing task-level information. Next, it uses a factor extraction network to produce causal generative factors for all tasks and a weight extraction network to assign dedicated weights to each sample, employing data reconstruction, orthogonality, and sparsity to ensure effectiveness. Finally, TC$^2$ calibrates sample representations during SSL training and integrates into the pipeline via a two-stage bi-level optimization framework to boost the transferability of learned representations. Experimental results on multiple downstream tasks demonstrate that our method consistently improves the transferability of SSL models.

Figures (6)

• Figure 1: Effect of task-level information on CIFAR-100. (a) shows the effect of SimCLR trained on ImageNet and tested on CIFAR-100 before and after introducing task-level information in five runs. (b) and (c) show the training process corresponding to the worst and best rounds in five evaluations, visualizing the gradient similarity between tasks.
• Figure 2: (a) Overview of the SCM based on generation mechanisms; (b) The causal mechanism of task conflict.
• Figure 3: Overview of the proposed framework.
• Figure 4: Model training pipeline for the motivational experiment. The encoder is jointly optimized based on two tasks.
• Figure 5: Effect of task-level information. It shows the effect of SimCLR trained on ImageNet and tested on three different datasets before and after introducing task-level information in five runs.

Theorems & Definitions (3)

• Theorem 1
• Theorem 1
• Proof 1