A Generic Self-Supervised Framework of Learning Invariant Discriminative Features

TL;DR

A generic SSL framework based on a constrained self-labeling assignment process that prevents degenerate solutions is proposed and outperforms a majority of state-of-the-art representation learning methods based on AE structures.

Abstract

Self-supervised learning (SSL) has become a popular method for generating invariant representations without the need for human annotations. Nonetheless, the desired invariant representation is achieved by utilising prior online transformation functions on the input data. As a result, each SSL framework is customised for a particular data type, e.g., visual data, and further modifications are required if it is used for other dataset types. On the other hand, autoencoder (AE), which is a generic and widely applicable framework, mainly focuses on dimension reduction and is not suited for learning invariant representation. This paper proposes a generic SSL framework based on a constrained self-labelling assignment process that prevents degenerate solutions. Specifically, the prior transformation functions are replaced with a self-transformation mechanism, derived through an unsupervised training process of adversarial training, for imposing invariant representations. Via the self-transformation mechanism, pairs of augmented instances can be generated from the same input data. Finally, a training objective based on contrastive learning is designed by leveraging both the self-labelling assignment and the self-transformation mechanism. Despite the fact that the self-transformation process is very generic, the proposed training strategy outperforms a majority of state-of-the-art representation learning methods based on AE structures. To validate the performance of our method, we conduct experiments on four types of data, namely visual, audio, text, and mass spectrometry data, and compare them in terms of four quantitative metrics. Our comparison results indicate that the proposed method demonstrate robustness and successfully identify patterns within the datasets.

Figures (7)

• Figure 1: This diagram illustrates our proposed framework with the two implemented modules. The encoder part, denoted by ($f_\psi$), projects the input ($x_i$), augmented by VAT perturbations ($r_{i,(vadv)}$), onto the latent representation ($z_{i,(vadv)}$). The classifier module ($g_\phi$) outputs a probabilistic output ($p_{i,(vadv)}$). Lastly, based on the computed logit predictions ($o_{i,(vadv)}$), model predictions prior to the Softmax function), the Sinkhorn-Knopp NIPS2013_af21d0c9 algorithm generates the desired target distribution ($q_{i,(vadv)}$). Note that in the diagram, the prime superscript index (e.g., ${p'}_{i,(vadv)}$) denotes the second augmented instance by the VAT process from the same data point ($x_i$)
• Figure 2: An illustration of the distribution of the probability mass of a single prediction $p_i^{(s)}$ at step $s$, and the corresponding distribution of the target $q_i^{(s)}$ (computed through the Sinkhorn-Knopp algorithm NIPS2013_af21d0c9), by varying the value of parameter $\lambda$. Note that the target distributions are computed from the same mini-batch prediction $\mathbf{P}^{(s)}$.
• Figure 3: The plot shows the value of the regularised entropy term $H(\textbf{Q^{(s)}})$, at $s$ training step, by varying $\lambda$ over a wide interval. The target $\textbf{Q}^{(s)}$ is computed from the same predictions across the range of $\lambda$ (x-axis). For clarity, the orange horizontal line indicates the entropy value of the original $\mathbf{P}^{(s)}$.
• Figure 4: Four scatter plots of our proposed framework and the AAE on the FSDD dataset are shown in this diagram. The ground truth labels of the matching digits are used to colourize the top line charts. The bottom line shows the same scatters coloured by the labels of the speakers.
• Figure 5: This figure presents the impact of the output dimensions $k$ ($x$-axis) in terms of 1) the accuracy left ($y$-axis) of the model in linear classification evaluation among the comparison dataset and 2) the RSS (right ($y$-axis)) of the linear regression for MSI data only.