Publishing Neural Networks in Drug Discovery Might Compromise Training Data Privacy

TL;DR

This study uses membership inference attacks, a common method to assess privacy that is largely unexplored in the context of drug discovery, to examine neural networks for molecular property prediction in a black-box setting, revealing significant privacy risks across all evaluated datasets and neural network architectures.

Abstract

This study investigates the risks of exposing confidential chemical structures when machine learning models trained on these structures are made publicly available. We use membership inference attacks, a common method to assess privacy that is largely unexplored in the context of drug discovery, to examine neural networks for molecular property prediction in a black-box setting. Our results reveal significant privacy risks across all evaluated datasets and neural network architectures. Combining multiple attacks increases these risks. Molecules from minority classes, often the most valuable in drug discovery, are particularly vulnerable. We also found that representing molecules as graphs and using message-passing neural networks may mitigate these risks. We provide a framework to assess privacy risks of classification models and molecular representations. Our findings highlight the need for careful consideration when sharing neural networks trained on proprietary chemical structures, informing organisations and researchers about the trade-offs between data confidentiality and model openness.

Figures (5)

• Figure 1: Overview of our workflow to evaluate privacy risks of neural network for molecular property prediction. Two random, non-overlapping subsets are created from each dataset. One subset is transformed into the desired molecular representation and used to train a neural network, optimised through Bayesian hyperparameter tuning bergstra2011algorithms. We then apply membership inference attacks (\ref{['alg:MIA']}) to determine if chemical structures in the training data can be distinguished from those in the other subset. We evaluate this using two different attack implementations. This process is repeated 20 times for each dataset and molecular representation. We assess the results by analyzing true positive rates at fixed false positive rates, comparing them to random guessing, and examining the impact of the molecular representations.
• Figure 2: True positive rates for identifying training data molecules at a false positive rate of 0. The distributions of 20 experimental repetitions are shown for each representation and dataset, for both the likelihood ratio attack (LiRA) and the robust membership inference attack (RMIA). Distributions with significantly higher true positive rates than the baseline are indicated by red stars. A single star represents a p-value less than 0.05, two stars represent a p-value less than 0.01, and three stars represent a p-value less than 0.001. Training dataset sizes (total amount of positives) are: 859 molecules for the blood-brain barrier permeability dataset; 3,264 for the Ames mutagenicity prediction dataset; 48,837 for the DNA-encoded library enrichment dataset; and 137,853 for the hERG channel inhibition dataset.
• Figure 3: Classification performance of neural networks trained on different molecular representations in molecular property prediction tasks. The performance is measured as the area under the receiver operating characteristic curve (AUROC). The performance is displayed as the distribution over 20 experiment repetitions.
• Figure 4: Overlap between the sets of molecules identified by the likelihood ratio attack (LiRA) and the robust membership inference attack (RMIA). The percentage of possible overlap is defined as the proportion of molecules from the smaller set that are also present in the larger set. The less overlap exists between the attacks, the more information is gained when combining them. Overlap that was significantly higher than observed when randomly drawing two uncorrelated subsets is indicated by red stars. A single star represents a p-value less than 0.05, two stars represent a p-value less than 0.01, and three stars represent a p-value less than 0.001.
• Figure 5: Chemical structures identified using the likelihood ratio attack (LiRA) against a neural network model trained to predict whether molecules pass the blood-brain barrier. Molecules were represented using ECFP4s in this model. Structures that are from the minority class have the label 0 and are surrounded by a solid line. These structures correspond to molecules that cannot pass the blood-brain barrier. It was possible to identify 23 of the 859 training structures at an FPR of 0.