MoRA: On-the-fly Molecule-aware Low-Rank Adaptation Framework for LLM-based Multi-Modal Molecular Assistant

TL;DR

MoRA introduces a molecule-aware, on-the-fly low-rank adaptation framework that dynamically generates instance-specific weight updates to condition a frozen LLM on molecular graphs. The Molecule-Aware Weight Generator (MAW-Gen) converts graph embeddings into low-rank updates that are injected into the LLM's internal blocks, enabling structure-aware reasoning without degrading the model's general capabilities. Across chemical reaction prediction, retrosynthesis, molecule description, and property prediction, MoRA achieves state-of-the-art results while preserving core LLM knowledge and offering scalable, instance-specific molecular understanding. The work demonstrates a pragmatic path toward integrating rich molecular structure with large language models, with practical implications for drug discovery and molecular design.

Abstract

Effectively integrating molecular graph structures with Large Language Models (LLMs) is a key challenge in drug discovery. Most existing multi-modal alignment methods typically process these structures by fine-tuning the LLM or adding a static adapter simultaneously. However, these approaches have two main limitations: (1) it optimizes a shared parameter space across all molecular inputs, limiting the model's ability to capture instance-specific structural features; and (2) fine-tuning the LLM for molecular tasks can lead to catastrophic forgetting, undermining its general reasoning capabilities. In this paper, instead of static task-oriented adaptation, we propose an instance-specific parameter space alignment approach for each molecule on-the-fly. To this end, we introduce Molecule-aware Low-Rank Adaptation (MoRA) that produces a unique set of low-rank adaptation weights for each input molecular graph. These weights are then dynamically injected into a frozen LLM, allowing the model to adapt its reasoning to the structure of each molecular input, while preserving the LLM's core knowledge. Extensive experiments demonstrate that on key molecular tasks, such as chemical reaction prediction and molecular captioning, MoRA's instance-specific dynamic adaptation outperforms statically adapted baselines, including a 14.1% relative improvement in reaction prediction exact match and a 22% reduction in error for quantum property prediction. The code is available at https://github.com/jk-sounds/MoRA.

Figures (6)

• Figure 1: Catastrophic forgetting in vicuna-7B after domain-specific LoRA fine-tuning on Mol-Instruction datasets. Original model performance (blue) versus fine-tuned model (red) across general benchmarks.
• Figure 2: Comparison of molecular multimodal language modeling.
• Figure 3: The overall architecture of our proposed MoRA framework. It comprises a graph encoder, a Molecule-Aware Weight Generator (MAW-Gen), and a frozen Large Language Model. MoRA first encodes an input 2D molecular graph with the graph encoder, then converts the encoded graph into a set of low-rank weight matrices via the MAW-Gen. These weights are subsequently injected into the parameters of the frozen LLM. Finally, the instruction-following response is generated by the LLM, conditioned on both the text prompt and its newly injected, molecule-aware weights.
• Figure 4: Ablation study on the injection targets for the molecule-aware weights. Targets include the self-attention projections—query (q), key (k), value (v), output (o)—and the feed-forward network (f).
• Figure 5: Computational efficiency analysis as a function of molecular complexity. (a) Total GFLOPS and inference time scaling with the number of atoms. (b) The corresponding efficiency score, highlighting resource utilization.