Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Large Molecular Language Model via Relation-aware Multimodal Collaboration

Jinyoung Park, Minseong Bae, Jeehye Na, Hyunwoo J. Kim

TL;DR

This work tackles hallucination and modality underutilization in large molecular language models by introducing CoLLaMo, a transformer-based molecular assistant equipped with a modality-collaborative projector and relation-aware attention to jointly reason over 1D SELFIES, 2D molecular graphs, and 3D conformations. It formalizes a unified token space for molecular representations and leverages a two-stage training pipeline (molecule-language alignment followed by molecule-language instruction-tuning with LoRA) to align encoders with an LLM. To address evaluation gaps, the authors propose molecule-centric metrics CHARM and RCHARM for hallucination and an LLM-based GPT-4o judge for description quality, enabling more faithful assessments. Empirically, CoLLaMo achieves state-of-the-art results across molecule captioning, descriptive/computed property QA, motif counting, and IUPAC naming, while remaining robust when modalities are partially available, demonstrating strong generalization for multimodal molecular reasoning. Core objective is to maximize $p(oldsymbol{Y}|oldsymbol{M},oldsymbol{T})=igg(igprod_{i=1}^{K} pig(y_i|oldsymbol{M},oldsymbol{T},oldsymbol{Y}_{<i}ig)igg)$, where $oldsymbol{M}$ combines 1D/2D/3D representations into unified tokens.

Abstract

Large language models (LLMs) have demonstrated their instruction-following capabilities and achieved powerful performance on various tasks. Inspired by their success, recent works in the molecular domain have led to the development of large molecular language models (LMLMs) that integrate 1D molecular strings or 2D molecular graphs into the language models. However, existing LMLMs often suffer from hallucination and limited robustness, largely due to inadequate integration of diverse molecular modalities such as 1D sequences, 2D molecular graphs, and 3D conformations. To address these limitations, we propose CoLLaMo, a large language model-based molecular assistant equipped with a multi-level molecular modality-collaborative projector. The relation-aware modality-collaborative attention mechanism in the projector facilitates fine-grained and relation-guided information exchange between atoms by incorporating 2D structural and 3D spatial relations. Furthermore, we present a molecule-centric new automatic measurement, including a hallucination assessment metric and GPT-based caption quality evaluation to address the limitations of token-based generic evaluation metrics (i.e., BLEU) widely used in assessing molecular comprehension of LMLMs. Our extensive experiments demonstrate that our CoLLaMo enhances the molecular modality generalization capabilities of LMLMs, achieving the best performance on multiple tasks, including molecule captioning, computed property QA, descriptive property QA, motif counting, and IUPAC name prediction.

Improving Large Molecular Language Model via Relation-aware Multimodal Collaboration

TL;DR

This work tackles hallucination and modality underutilization in large molecular language models by introducing CoLLaMo, a transformer-based molecular assistant equipped with a modality-collaborative projector and relation-aware attention to jointly reason over 1D SELFIES, 2D molecular graphs, and 3D conformations. It formalizes a unified token space for molecular representations and leverages a two-stage training pipeline (molecule-language alignment followed by molecule-language instruction-tuning with LoRA) to align encoders with an LLM. To address evaluation gaps, the authors propose molecule-centric metrics CHARM and RCHARM for hallucination and an LLM-based GPT-4o judge for description quality, enabling more faithful assessments. Empirically, CoLLaMo achieves state-of-the-art results across molecule captioning, descriptive/computed property QA, motif counting, and IUPAC naming, while remaining robust when modalities are partially available, demonstrating strong generalization for multimodal molecular reasoning. Core objective is to maximize , where combines 1D/2D/3D representations into unified tokens.

Abstract

Large language models (LLMs) have demonstrated their instruction-following capabilities and achieved powerful performance on various tasks. Inspired by their success, recent works in the molecular domain have led to the development of large molecular language models (LMLMs) that integrate 1D molecular strings or 2D molecular graphs into the language models. However, existing LMLMs often suffer from hallucination and limited robustness, largely due to inadequate integration of diverse molecular modalities such as 1D sequences, 2D molecular graphs, and 3D conformations. To address these limitations, we propose CoLLaMo, a large language model-based molecular assistant equipped with a multi-level molecular modality-collaborative projector. The relation-aware modality-collaborative attention mechanism in the projector facilitates fine-grained and relation-guided information exchange between atoms by incorporating 2D structural and 3D spatial relations. Furthermore, we present a molecule-centric new automatic measurement, including a hallucination assessment metric and GPT-based caption quality evaluation to address the limitations of token-based generic evaluation metrics (i.e., BLEU) widely used in assessing molecular comprehension of LMLMs. Our extensive experiments demonstrate that our CoLLaMo enhances the molecular modality generalization capabilities of LMLMs, achieving the best performance on multiple tasks, including molecule captioning, computed property QA, descriptive property QA, motif counting, and IUPAC name prediction.
Paper Structure (15 sections, 7 equations, 2 figures, 7 tables)

This paper contains 15 sections, 7 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Impact of Molecular Modalities
  5. CoLLaMo
  6. Molecular modality-collaborative projector
  7. Training CoLLaMo
  8. Advancing Molecule-Aware Evaluation
  9. Hallucination assessment.
  10. LLM-based molecule description assessment.
  11. Experiments
  12. Experimental Settings
  13. Experimental Results
  14. Analysis
  15. Conclusion

Figures (2)

  • Figure 1: Overall architecture of CoLLaMo. Our framework, CoLLaMo, consists of encoders for 1D SELFIES, 2D graphs, and 3D conformations, a molecular modality-collaborative projector, and a large language model. Our framework first encodes 1D, 2D, and 3D molecular representations and then converts the encoded outputs into unified molecule tokens using the modality-collaborative projector. Finally, the large language model generates the response given the input molecule and instruction.
  • Figure 2: Examples of molecule captioning given the molecule (CHEBI:65102). The molecule structures of underlined texts in generated descriptions are illustrated in the white box.