nach0: Multimodal Natural and Chemical Languages Foundation Model

TL;DR

nach0 presents a multimodal foundation model that unifies natural language and chemical representations within a single encoder-decoder architecture. Trained with a text-to-text, instruction-tuned regime on large NL and chemical data using the NeMo framework, nach0 demonstrates competitive performance on both single-domain NLP/chemistry tasks and cross-domain tasks such as description-guided molecule design and retrosynthesis. The work provides extensive benchmarks across NLP datasets, MoleculeNet, and Mol-Instructions, showing strong generation, property prediction, and cross-domain reasoning capabilities, and it includes two practical case studies: end-to-end drug discovery and cross-domain molecule generation. While achieving notable gains, the authors acknowledge limitations in chemist-level reasoning, SMILES-only representations, and the need for future extensions to 3D molecular data and knowledge-graph integration to reach expert performance at scale.

Abstract

Large Language Models (LLMs) have substantially driven scientific progress in various domains, and many papers have demonstrated their ability to tackle complex problems with creative solutions. Our paper introduces a new foundation model, nach0, capable of solving various chemical and biological tasks: biomedical question answering, named entity recognition, molecular generation, molecular synthesis, attributes prediction, and others. nach0 is a multi-domain and multi-task encoder-decoder LLM pre-trained on unlabeled text from scientific literature, patents, and molecule strings to incorporate a range of chemical and linguistic knowledge. We employed instruction tuning, where specific task-related instructions are utilized to fine-tune nach0 for the final set of tasks. To train nach0 effectively, we leverage the NeMo framework, enabling efficient parallel optimization of both base and large model versions. Extensive experiments demonstrate that our model outperforms state-of-the-art baselines on single-domain and cross-domain tasks. Furthermore, it can generate high-quality outputs in molecular and textual formats, showcasing its effectiveness in multi-domain setups.

Figures (4)

• Figure 1: A Venn diagram that shows the relationships between fine-tuning data used in our study and related work. It is important to highlight that the majority of models typically treat the chemical space and the semantic space in the natural language domain independently. Novel cross-domain datasets such as Mol-Instructions fang2023mol and MolT5 data edwards2022translation have asked whether it is possible to unify representations of natural language and molecules for NLP and molecule generation tasks within a single model. In this work, we seek to answer this question.
• Figure 2: Datasets used for training and evaluation. Colour represents the type of tasks. Yellow and blue datasets are single-domain, typically requiring regression/classification losses or generation in the target domain (natural language or SMILES strings). Gradients from yellow to blue represent cross-domain generation tasks that require natural language input and SMILES output, or vise versa.
• Figure 3: A diagram of nach0 which is a text-to-text framework. The model takes text as input and is trained to generate the desired target text for each specific task. This unified approach enables us to utilize the same model architecture, loss function, hyperparameters, and other components across our diverse range of mono-domain (NLP, CHEM) and cross-domain (NLP$\leftrightarrow$CHEM) tasks.
• Figure 4: Input request from a human (gray color) and nach0's response (blue color).