Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Tuning Vision-Language Models for Multimodal Polymer Property Prediction

An Vuong, Minh-Hao Van, Prateek Verma, Chen Zhao, Xintao Wu

TL;DR

The paper tackles the challenge of predicting polymer properties from multimodal data by developing a multimodal polymer dataset and fine-tuning Vision-Language Models via instruction-tuning with LoRA. It demonstrates that a unified, multimodal approach using images, P-SMILES, and descriptors can outperform unimodal and baseline methods while reducing the need for separate per-property models. Across Kaggle and unseen datasets, fine-tuned LVision and related models achieve competitive or superior performance, especially for Density prediction, highlighting the value of visual information in polymer tasks. This work suggests a practical path toward foundation-model-style multimodal reasoning in materials science and points to future work in expanding data and modalities.

Abstract

Vision-Language Models (VLMs) have shown strong performance in tasks like visual question answering and multimodal text generation, but their effectiveness in scientific domains such as materials science remains limited. While some machine learning methods have addressed specific challenges in this field, there is still a lack of foundation models designed for broad tasks like polymer property prediction using multimodal data. In this work, we present a multimodal polymer dataset to fine-tune VLMs through instruction-tuning pairs and assess the impact of multimodality on prediction performance. Our fine-tuned models, using LoRA, outperform unimodal and baseline approaches, demonstrating the benefits of multimodal learning. Additionally, this approach reduces the need to train separate models for different properties, lowering deployment and maintenance costs.

Fine-Tuning Vision-Language Models for Multimodal Polymer Property Prediction

TL;DR

The paper tackles the challenge of predicting polymer properties from multimodal data by developing a multimodal polymer dataset and fine-tuning Vision-Language Models via instruction-tuning with LoRA. It demonstrates that a unified, multimodal approach using images, P-SMILES, and descriptors can outperform unimodal and baseline methods while reducing the need for separate per-property models. Across Kaggle and unseen datasets, fine-tuned LVision and related models achieve competitive or superior performance, especially for Density prediction, highlighting the value of visual information in polymer tasks. This work suggests a practical path toward foundation-model-style multimodal reasoning in materials science and points to future work in expanding data and modalities.

Abstract

Vision-Language Models (VLMs) have shown strong performance in tasks like visual question answering and multimodal text generation, but their effectiveness in scientific domains such as materials science remains limited. While some machine learning methods have addressed specific challenges in this field, there is still a lack of foundation models designed for broad tasks like polymer property prediction using multimodal data. In this work, we present a multimodal polymer dataset to fine-tune VLMs through instruction-tuning pairs and assess the impact of multimodality on prediction performance. Our fine-tuned models, using LoRA, outperform unimodal and baseline approaches, demonstrating the benefits of multimodal learning. Additionally, this approach reduces the need to train separate models for different properties, lowering deployment and maintenance costs.

Paper Structure

This paper contains 18 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminary
  4. Large-Language Models
  5. Vision-Language Models
  6. Fine-tuning Vision-Language Models for Multimodal Polymer Property Prediction
  7. Multimodal Polymer Dataset
  8. Data Collection and Preprocessing
  9. Multimodal Features Generation
  10. Instruction-tuning Dataset Generation
  11. Fine-tuning VLMs with LoRA
  12. Experiments
  13. Models
  14. Metrics
  15. Results and Discussion
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Pipeline for multimodal prompt generation.
  • Figure 2: An example of our instruction-tuning sample.
  • Figure 3: An illustrative figure of fine-tuning VLMs with LoRA. Parameters of Transformer blocks in LLM and Vision Encoder are decomposed into two low-rank matrices that will be updated.