Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning

TL;DR

This work presents model reprogramming as a resource-efficient strategy to repurpose frozen, pre-trained source models for cross-domain tasks by adding lightweight input-transform and output-mapping adapters. It formalizes a general framework, details concrete continuous and discrete data implementations, and provides a theoretical bound linking target risk to source risk and representation alignment. The paper surveys multiple use cases, including BAR, V2S, R2DL, and WARP, illustrating data-efficient performance in data-scarce settings. It also discusses open questions, from semi-supervised extensions to foundation-model reprogramming and deployment in heterogeneous computing environments, highlighting practical pathways to democratize access to powerful pre-trained models.

Abstract

In data-rich domains such as vision, language, and speech, deep learning prevails to deliver high-performance task-specific models and can even learn general task-agnostic representations for efficient finetuning to downstream tasks. However, deep learning in resource-limited domains still faces multiple challenges including (i) limited data, (ii) constrained model development cost, and (iii) lack of adequate pre-trained models for effective finetuning. This paper provides an overview of model reprogramming to bridge this gap. Model reprogramming enables resource-efficient cross-domain machine learning by repurposing and reusing a well-developed pre-trained model from a source domain to solve tasks in a target domain without model finetuning, where the source and target domains can be vastly different. In many applications, model reprogramming outperforms transfer learning and training from scratch. This paper elucidates the methodology of model reprogramming, summarizes existing use cases, provides a theoretical explanation of the success of model reprogramming, and concludes with a discussion on open-ended research questions and opportunities. A list of model reprogramming studies is actively maintained and updated at https://github.com/IBM/model-reprogramming.

Figures (2)

• Figure 1: Visual illustration of data scale (bottom to top: small to large) and number (#) of trainable parameters (left to right: small to large) in different machine learning paradigms. We note that the visualization does not reflect the actual relative differences due to excessively varying orders. A foundation model like GPT-3 has 175 billion trainable parameters and 499 billion tokens as training data. The trainable parameters in model reprogramming can be as few as the size of the data input (e.g., the number of image pixels can be in the order of thousands or fewer), and model reprogramming is particularly suited to small-scale data regime. In model reprogramming, the visualization does not take into account the pre-trained source model because it is kept intact and unchanged. The dashed box in transfer learning means variations in the number of model parameters used for fine-tuning, ranging from only training the last dense layer (linear head) to fine-tuning all parameters. The number of training epochs may also vary for each paradigm.
• Figure 2: Illustration of the model reprogramming framework (top) and some examples of cross-domain machine learning via model reprogramming (bottom). Model reprogramming enables cross-domain machine learning by adding two modules, an input transformation layer (blue box) and an output mapping layer (green box), to a pre-trained model selected from a source domain. When reprogrammed to solve target-domain tasks, the pre-trained source model is frozen and its model parameters are unchanged. Examples of cross-domain machine learning include reprogramming speech models for time-series yang2021voice2series, language models for molecules vinod2023reprogramming, and general imaging models for bio-medical measurements tsai2020transfer.