ES-Merging: Biological MLLM Merging via Embedding Space Signals

Abstract

Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery. However, existing models are specialized to a single modality, limiting their ability to solve inherently cross-modal scientific problems. While model merging is an efficient method to combine the different modalities into a unified MLLM, existing methods rely on input-agnostic parameter space heuristics that fail to faithfully capture modality specialization. To overcome this limitation, we propose a representation-aware merging framework that estimates merging coefficients from embedding space signals. We first design a probe input that consists of different modality tokens and forward it through each specialized MLLM to obtain layer-wise embedding responses that reflect modality-specific representation changes. We then estimate complementary merging coefficients at two granularities from the embedding space: layer-wise coefficients from coarse-grained signals and element-wise coefficients from fine-grained signals, which are jointly combined for robust coefficient estimation. Experiments on interactive effect prediction benchmarks show that our method outperforms existing merging methods and even surpasses task-specific fine-tuned models, establishing that embedding space signals provide a principled and effective foundation for cross-modal MLLM merging.

Figures (15)

• Figure 1: Molecule token embedding visualization of the last transformer block for the base LLM and each specialized LLM.
• Figure 2: Layer-wise embedding distribution distance under specialized and non-specialized tokens using sliced Wasserstein distances (SWD) swd.
• Figure 3: Overview of ES-Merging. (A) Layer-wise global merging coefficients are computed from the coarse-grained embedding signals, which are the layer-wise differences of distribution distances between mean pooled representations of the base LLM and a specialized MLLM. (B) Element-wise local merging coefficients are assigned based on the fine-grained embedding signals by computing the gradients from the embedding-wise distances.
• Figure 4: Prompt template of the probe input.
• Figure 5: Computed layer-wise merging coefficient visualization of each specialized MLLM derived by Eq. \ref{['eq:layer_wise_merging_coefficient']}. The $l$-th column corresponds to the merging coefficient of the $l$-th layer, $\alpha^l_{m_j}$.