Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network

Bin Cheng, Jiaxuan Lu

TL;DR

This work introduces GA-Net, a graph-based, parameter-efficient fine-tuning framework for multi-modal tasks that freezes the backbone while learning via a Graph Adapter Network over a multi-modal feature graph. By generating text with an MLLM, encoding image and text with frozen encoders, and wiring their features into a thresholded similarity graph, GA-Net harnesses cross-modal relationships through a bottleneck GCN. The approach, enhanced with Elastic Weight Consolidation to mitigate forgetting, achieves state-of-the-art gains on three fine-grained datasets while maintaining minimal trainable parameters and competitive memory usage. This graph-based PEFT method offers a practical path to efficient, scalable multi-modal adaptation in large foundation models.

Abstract

With the advent of the era of foundation models, pre-training and fine-tuning have become common paradigms. Recently, parameter-efficient fine-tuning has garnered widespread attention due to its better balance between the number of learnable parameters and performance. However, some current parameter-efficient fine-tuning methods only model a single modality and lack the utilization of structural knowledge in downstream tasks. To address this issue, this paper proposes a multi-modal parameter-efficient fine-tuning method based on graph networks. Each image is fed into a multi-modal large language model (MLLM) to generate a text description. The image and its corresponding text description are then processed by a frozen image encoder and text encoder to generate image features and text features, respectively. A graph is constructed based on the similarity of the multi-modal feature nodes, and knowledge and relationships relevant to these features are extracted from each node. Additionally, Elastic Weight Consolidation (EWC) regularization is incorporated into the loss function to mitigate the problem of forgetting during task learning. The proposed model achieves test accuracies on the OxfordPets, Flowers102, and Food101 datasets that improve by 4.45%, 2.92%, and 0.23%, respectively. The code is available at https://github.com/yunche0/GA-Net/tree/master.

Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network

TL;DR

This work introduces GA-Net, a graph-based, parameter-efficient fine-tuning framework for multi-modal tasks that freezes the backbone while learning via a Graph Adapter Network over a multi-modal feature graph. By generating text with an MLLM, encoding image and text with frozen encoders, and wiring their features into a thresholded similarity graph, GA-Net harnesses cross-modal relationships through a bottleneck GCN. The approach, enhanced with Elastic Weight Consolidation to mitigate forgetting, achieves state-of-the-art gains on three fine-grained datasets while maintaining minimal trainable parameters and competitive memory usage. This graph-based PEFT method offers a practical path to efficient, scalable multi-modal adaptation in large foundation models.

Abstract

With the advent of the era of foundation models, pre-training and fine-tuning have become common paradigms. Recently, parameter-efficient fine-tuning has garnered widespread attention due to its better balance between the number of learnable parameters and performance. However, some current parameter-efficient fine-tuning methods only model a single modality and lack the utilization of structural knowledge in downstream tasks. To address this issue, this paper proposes a multi-modal parameter-efficient fine-tuning method based on graph networks. Each image is fed into a multi-modal large language model (MLLM) to generate a text description. The image and its corresponding text description are then processed by a frozen image encoder and text encoder to generate image features and text features, respectively. A graph is constructed based on the similarity of the multi-modal feature nodes, and knowledge and relationships relevant to these features are extracted from each node. Additionally, Elastic Weight Consolidation (EWC) regularization is incorporated into the loss function to mitigate the problem of forgetting during task learning. The proposed model achieves test accuracies on the OxfordPets, Flowers102, and Food101 datasets that improve by 4.45%, 2.92%, and 0.23%, respectively. The code is available at https://github.com/yunche0/GA-Net/tree/master.
Paper Structure (16 sections, 11 equations, 3 figures, 3 tables)

This paper contains 16 sections, 11 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Multi-Modal Feature Extraction
  5. Multi-Modal Graph Construction
  6. Graph Adapter Net
  7. Prediction
  8. Experiment
  9. Experiment Settings
  10. Implementation Details
  11. Comparisons with State-of-the-Arts
  12. Model Efficiency
  13. Ablation Study
  14. Hyperparameter Study
  15. Conclusions
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Overall pipeline of the proposed framework. The proposed model consists of four main modules: Multi-Modal Feature Extraction, Multi-Modal Graph Construction, GA-Net, and Prediction. GA-Net updates vertex features through down, GCN, and up operations, while the Prediction module uses a combined EWC and cross-entropy loss function to improve performance.
  • Figure 2: Details of GA-Net. By projecting the graph structure downwards, the number of model parameters is significantly reduced. Then, a GCN network aggregates information from adjacent nodes to learn complex associations between different modalities. Finally, the structure is projected upwards, restoring the original graph size.
  • Figure 3: Accuracy under different similarity thresholds