Rethinking Cross-Modal Fine-Tuning: Optimizing the Interaction between Feature Alignment and Target Fitting

TL;DR

This work tackles cross-modal fine-tuning by deriving a provable generalization bound that separates the target error into an overhead from the source model, feature alignment, feature-label distortion, and target fitting: $err_\tau(\phi) \leq err_s(\theta) + FA(\phi,\theta) + \mathbb{E}_{D^{\phi}_{\tau}(\boldsymbol{u})}[FLD(\boldsymbol{u}) + TF(\boldsymbol{u})]$. It introduces RECRAFT, a two-stage algorithm that first learns a target feature map by minimizing the semantic gap via surrogates $L_{\textbf{FA}}(\phi)$ and $L_{\textbf{FLD}}(\phi)$ and then learns a target predictor to minimize the target-fitting term $TF$, leveraging probabilistic transport maps between source and target distributions. Theoretical insights are complemented by extensive empirical validation on NAS-Bench-360 and PDEBench, where RECRAFT consistently outperforms state-of-the-art baselines (e.g., ORCA, PARE, MoNA) and exhibits a tight bound in practice. Overall, the paper provides a principled framework for cross-modal knowledge transfer that accounts for semantic alignment and label-level transferability, enabling more reliable generalization across diverse modalities.

Abstract

Adapting pre-trained models to unseen feature modalities has become increasingly important due to the growing need for cross-disciplinary knowledge integration.~A key challenge here is how to align the representation of new modalities with the most relevant parts of the pre-trained model's representation space to enable accurate knowledge transfer.~This requires combining feature alignment with target fine-tuning, but uncalibrated combinations can exacerbate misalignment between the source and target feature-label structures and reduce target generalization.~Existing work however lacks a theoretical understanding of this critical interaction between feature alignment and target fitting.~To bridge this gap, we develop a principled framework that establishes a provable generalization bound on the target error, which explains the interaction between feature alignment and target fitting through a novel concept of feature-label distortion.~This bound offers actionable insights into how this interaction should be optimized for practical algorithm design. The resulting approach achieves significantly improved performance over state-of-the-art methods across a wide range of benchmark datasets.

Figures (7)

• Figure 1: Overview of the RECRAFT algorithm.
• Figure 2: Visualizations of representation alignment (via tSNE) under $3$ settings: (a) naive fine-tuning (NFT), which ignores alignment; (b) minimization of feature alignment ($\textbf{FA}$) via Eq. \ref{['eq:FA-def']}; and (c) minimizing a sum of $\textbf{FA}$ and feature-label distortion ($\textbf{FLD}$) via Eq. \ref{['eq:feature']}. The corresponding predictive errors are shown in (d). NFT exhibits no alignment while minimizing $\textbf{FA}$ leads to exhaustive alignment. Both results in suboptimal performance. In contrast, minimizing $\textbf{FA} + \textbf{FLD}$ enables selective alignment and achieves the best performance.
• Figure 3: Target error versus semantic gap (Eq. \ref{['eq:feature']}) for RECRAFT on ECG, NinaPro, and DeepSEA. All plots show a strong, consistent correlation across tasks.
• Figure 4: Bar charts illustrating the gap between the target's generalization loss and its upper-bound established in Theorem \ref{['thm:2']}. For each target task, there are two bars representing the target's generalization loss and its upper-bound. The bar representing the upper-bound is further demarcated into the source's generalization loss (fine-tuning overhead), feature alignment(FA), feature Label distortion(FLD), and target fitting(TF). The bound evaluation is conducted at the learned target's feature map $\phi$ and its corresponding prediction head $p_\tau(z'\mid\boldsymbol{u})$.
• Figure 5: Predictive error (↓) of Darcy Flow, Cosmic and Ninapro across various value of $\omega$, which achieves the balance between minimizing Feature Alignment(FA) and Feature Label Distortion(FLD).

Theorems & Definitions (7)

• Definition 1: Generalized Error
• Definition 2: Feature Distribution
• Theorem 3: Informal Statement
• Definition 4: Feature Alignment
• Definition 5: Feature-Label Distortion
• Definition 6: Target Fitting
• Theorem 7: Formal Statement