Pretrained Generative Language Models as General Learning Frameworks for Sequence-Based Tasks

TL;DR

This work shows that small pretrained generative language models (millions of parameters) can serve as general learning frameworks for sequence tasks via instruction fine-tuning on domain-specific data, offering substantial reductions in training resources and development time. On a chemistry-centric task—translating SMILES strings to IUPAC names—125M–1.3B parameter models achieve near-state-of-the-art results when fine-tuned with 10,000 to 1,000,000 instruction examples, with performance strongly influenced by model size and data formatting. The study systematically explores data size, fine-tuning epochs, model architectures (including TinyStories), and adapter-based methods, highlighting that formatting the instruction data to match the task is crucial and that larger models and more epochs generally help, though returns diminish after ~20–30 epochs. The results suggest a practical path to deploying specialized, resource-efficient language-model learners across domains, while acknowledging that rules-based approaches may still outperform these methods for certain cheminformatics tasks.

Abstract

We propose that small pretrained foundational generative language models with millions of parameters can be utilized as a general learning framework for sequence-based tasks. Our proposal overcomes the computational resource, skill set, and timeline challenges associated with training neural networks and language models from scratch. Further, our approach focuses on creating small and highly specialized models that can accurately execute a challenging task of which the base model is incapable of performing. We demonstrate that 125M, 350M, and 1.3B parameter pretrained foundational language models can be instruction fine-tuned with 10,000-to-1,000,000 instruction examples to achieve near state-of-the-art results on challenging cheminformatics tasks. We also demonstrate the role of successive language model fine-tuning epochs on improved outcomes, as well as the importance of both data formatting and pretrained foundational language model selection for instruction fine-tuning success.

Figures (9)

• Figure 1: Illustration of our proposed task: convert a molecular SMILES string into the corresponding IUPAC chemical name. Skeletal structure of caffeine with the SMILES string representation and corresponding IUPAC chemical name are shown as an example.
• Figure 2: Influence of increasing instruction fine-tuning examples. Mean normalized edit distance versus count of the instruction fine-tuning examples for the OPT-125M (blue) and OPT-1.3B (orange) language models, instruction fine-tuned with learning rate = 2e-5, batch size = 4, and epochs = 3. Performance of the instruction fine-tuned language models were assessed with 1,000 test instances of SMILES strings, and the model’s ability to accurately generate the corresponding IUPAC chemical names relative to the ground truth.
• Figure 3: Influence of increasing instruction fine-tuning epochs. Mean normalized edit distance versus instruction fine-tuning epochs for 10,000 (blue) and 100,000 (orange) instruction fine-tuning examples and the OPT-125M pretrained foundational language model, instruction fine-tuned with learning rate = 2e-5 and batch size = 4. Performance of the instruction fine-tuned language models were assessed with 1,000 test instances of SMILES strings, and the model’s ability to accurately generate the corresponding IUPAC chemical names relative to the ground truth.
• Figure 4: Influence of increasing instruction fine-tuning epochs on % exact matches. Percentage of exact matches versus instruction fine-tuning epochs for 10,000 (blue) and 100,000 (orange) instruction fine-tuning examples and the OPT-125M pretrained foundational language model, instruction fine-tuned with learning rate = 2e-5 and batch size = 4. Performance of the instruction fine-tuned language models were assessed with 1,000 test instances of SMILES strings, and the model’s ability to accurately generate the corresponding IUPAC chemical names relative to the ground truth.
• Figure 5: Influence of increasing instruction fine-tuned language model parameter count on % exact matches. Percentage of exact matches versus 100,000 and 1,000,000 instruction fine-tuning examples for the OPT-125M (orange), OPT-350M (blue), and OPT-1.3B (grey) pretrained foundational language models, instruction fine-tuned with learning rate = 2e-5, batch size = 4, and epochs = 3. Performance of the instruction fine-tuned language models were assessed with 1,000 test instances of SMILES strings, and the model’s ability to accurately generate the corresponding IUPAC chemical names relative to the ground truth.