Molecular De Novo Design through Transformer-based Reinforcement Learning
Pengcheng Xu, Tao Feng, Tianfan Fu, Siddhartha Laghuvarapu, Jimeng Sun
TL;DR
This work tackles de novo molecular design by replacing RNNs with Transformer-based generators and optimizing them via oracle-guided reinforcement learning (RL). The REINVENT-Transformer framework pre-trains a Transformer on SMILES-encoded 2D molecules using maximum likelihood, then fine-tunes it with policy-based RL that leverages an oracle to reward desirable properties, combining prior likelihood with task-specific scores. Empirical results across the ZINC 250K dataset and multiple oracles show the Transformer-based approach delivers superior performance, particularly for long molecular sequences, faster convergence, and strong results on scaffold hopping and other targeted attributes. The findings highlight the potential of Transformer architectures, when coupled with oracle feedback, to advance practical, property-driven molecular design for drug discovery.
Abstract
In this work, we introduce a method to fine-tune a Transformer-based generative model for molecular de novo design. Leveraging the superior sequence learning capacity of Transformers over Recurrent Neural Networks (RNNs), our model can generate molecular structures with desired properties effectively. In contrast to the traditional RNN-based models, our proposed method exhibits superior performance in generating compounds predicted to be active against various biological targets, capturing long-term dependencies in the molecular structure sequence. The model's efficacy is demonstrated across numerous tasks, including generating analogues to a query structure and producing compounds with particular attributes, outperforming the baseline RNN-based methods. Our approach can be used for scaffold hopping, library expansion starting from a single molecule, and generating compounds with high predicted activity against biological targets.