PolyCL: Contrastive Learning for Polymer Representation Learning via Explicit and Implicit Augmentations

TL;DR

PolyCL tackles the challenge of learning generalizable polymer representations from unlabeled data by combining explicit and implicit augmentations in a contrastive learning framework built on a pre-trained PolyBERT encoder. It uses a NT-Xent objective with a carefully designed augmentation strategy to produce two informative views of each polymer-SMILES input, yielding 600-d representations that serve effectively as a frozen feature extractor for seven downstream polymer properties. Across extensive ablations and comparisons to supervised baselines and other self-supervised models, PolyCL achieves top-tier transfer performance, with significant gains on ionisation energy, dielectric constant, and refractive index, while maintaining robustness across tasks. The work highlights the importance of augmentations in contrastive polymer learning and provides a practical, scalable pathway to high-quality, task-agnostic polymer representations for informatics pipelines.

Abstract

Polymers play a crucial role in a wide array of applications due to their diverse and tunable properties. Establishing the relationship between polymer representations and their properties is crucial to the computational design and screening of potential polymers via machine learning. The quality of the representation significantly influences the effectiveness of these computational methods. Here, we present a self-supervised contrastive learning paradigm, PolyCL, for learning high-quality polymer representation without the need for labels. Our model combines explicit and implicit augmentation strategies for improved learning performance. The results demonstrate that our model achieves either better, or highly competitive, performances on transfer learning tasks as a feature extractor without an overcomplicated training strategy or hyperparameter optimisation. Further enhancing the efficacy of our model, we conducted extensive analyses on various augmentation combinations used in contrastive learning. This led to identifying the most effective combination to maximise PolyCL's performance.

Figures (5)

• Figure 1: Example of (a) SMILES of vinyl chloride and (b) polymer-SMILES of polyvinyl chloride, along with the corresponding chemical structures.
• Figure 2: A schematic illustration of the PolyCL pipeline. (1) Polymer contrastive representation learning with different augmentation strategies for constructing effective positive pairs. The agreement of positive pairs projected to their latent representations is maximised by the loss function of contrastive learning. Masking and Drop in augmented views 1 and 2 are shown as sample explicit augmentations for the input original polymer-SMILES. (2) Transfer learning by leveraging the acquired polymer representation to apply in the prediction of downstream tasks.
• Figure 3: Predictive performance of transfer learning evaluated by $R^2$ values on downstream datasets using contrastive learning trained with different augmentation combinations. (a) Explicit augmentations only (where "Enum" refers to Enumeration) (b) Implicit and selected mixed augmentation strategy. The striped background cells are the results using the contrastive learning model pretrained with no augmentation (the baseline result). Blue blocks show improved performance relative to the baseline. Red blocks show decreased performance relative to the baseline. The intensity of the colour reflects the magnitude of the deviation.
• Figure 4: Cross-model comparison on the alignment-uniformity space. For PolyBERT and Transpolymer, the alignment and uniformity of only the final published model is shown. For PolyCL and PolyCL with different augmentation combinations, the intermediate progress during contrastive pre-training is recorded and evaluated with alignment and uniformity. The coloured arrows denote the direction of change during training. The axis label arrows denote the favourable direction.
• Figure 5: t-SNE dimensional reduction analysis of the polymer representation space learnt by PolyCL. Visualisation of the continuous representation of polymer repeating units: (a) The unsupervised pretrained dataset coloured by molecular weight; (b) The Egc dataset coloured by the band gap (chain) property and (c) all available datasets coloured by the data origin, with selected polymers shown. The blue dot denotes the connection point of the repeating unit to the polymer chain.