Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction

TL;DR

This survey introduces neural network reprogrammability as a unifying framework that reconciles model reprogramming, prompt tuning, and prompt instruction under a single interface-based paradigm. It formalizes input manipulation and output alignment over model interfaces, offering a modality-agnostic taxonomy with four axes (manipulation format, location, operator, and output alignment). By recasting MR, PT, and PI as instantiations of RCA, the work clarifies connections, underlying mechanisms, and the trade-offs between fixed versus learnable prompts, and shallow versus deep interface interventions. It also discusses emergent insights on in-context learning and chain-of-thought, outlines evaluation challenges, and highlights ethical considerations, setting a path toward scalable, efficient, and responsible adaptation of large foundation models.

Abstract

As large-scale pre-trained foundation models continue to expand in size and capability, efficiently adapting them to specific downstream tasks has become increasingly critical. Despite substantial progress, existing adaptation approaches have evolved largely in isolation, without a clear understanding of their interrelationships. This survey introduces neural network reprogrammability as a unifying framework that bridges mainstream model adaptation techniques--model reprogramming, prompt tuning, and prompt instruction--previously fragmented research areas yet converges on a shared principle: repurposing a pre-trained model by manipulating information at the interfaces while keeping the model parameters frozen. These methods exploit neural networks' sensitivity to manipulation on different interfaces, be it through perturbing inputs, inserting tokens into intermediate layers, or providing task-specific examples in context, to redirect model behaviors towards desired outcomes. We then present a taxonomy that categorizes such information manipulation-based adaptation approaches across four key dimensions: manipulation format (fixed or learnable), location (interfaces where manipulations occur), operator (how they are applied), and output alignment requirement (post-processing needed to align outputs with downstream tasks). Notably, this framework applies consistently across data modalities, independent of specific model architectures. Moreover, viewing established techniques like in-context learning and chain-of-thought prompting through this lens reveals both their theoretical connections and practical distinctions. We further analyze remaining technical challenges and ethical considerations, positioning neural network reprogrammability as a fundamental paradigm for efficient model adaptation. We lastly identify promising research directions emerging from this integrative viewpoint.

Figures (5)

• Figure 1: Paradigm shift from conventional parameter-centric adaptation (i.e., modifying model parameters) to reprogrammability-centric adaptation (i.e., modifying input data and model output). This represents a shift in thought from modifying the model to align with the task to modifying the task to align with the model.
• Figure 2:
• Figure 3: We introduce Neural Network Reprogrammability as a unifying framework to bring coherence to a set of model adaptation techniques that have often been studied in isolation. Importantly, this shared underlying principle applies regardless of pre-trained model architectures and data modalities.
• Figure 4: Comparisons of implementations between MR, PT, and PI (using a pre-trained vision-language model as the example), distinguishing from each other by different manipulation forms, locations, operators, and output alignment strategies.
• Figure 5: Examples of how reprogrammability manifests across different adaptation methodologies, data modalities, and downstream tasks. (a) MR that repurposes a pre-trained image classifier for a new image classification task. (b) PT that repurposes a pre-trained language generator for a new text generation task.

Theorems & Definitions (5)

• Definition 1: Neural Network Reprogrammability
• Remark 1
• Remark 2
• Remark 3
• Remark 4