FUSE-ing Language Models: Zero-Shot Adapter Discovery for Prompt Optimization Across Tokenizers

TL;DR

The paper tackles interoperability across language and vision-language models that use different tokenizers and embedding spaces, which hampers cross-model prompt discovery. It introduces FUSE, a lightweight adapter that approximates a cross-model mapping between embedding spaces using a novel third-order tensor representation of whitespace-delimited token groups and a tensor-based t-product framework to enable gradient-like signals across diverse tokenizations. The method supports zero-shot task composition by enabling multi-model objective optimization (e.g., captioning with image and sentiment objectives) without fine-tuning, demonstrated on image captioning benchmarks with GPT2-Medium, CLIP, and a sentiment classifier. Findings indicate that the FUSE gradient approximation yields meaningful guidance for prompt optimization—especially improving SPICE scores—while remaining compute-efficient and suitable for low-resource settings, with future work aimed at training explicit cross-embedding mappings and improving scalability and validation of the gradient signal.

Abstract

The widespread use of large language models has resulted in a multitude of tokenizers and embedding spaces, making knowledge transfer in prompt discovery tasks difficult. In this work, we propose FUSE (Flexible Unification of Semantic Embeddings), an inexpensive approach to approximating an adapter layer that maps from one model's textual embedding space to another, even across different tokenizers. We introduce a third-order tensor-based representation of a model's embedding space that aligns semantic embeddings that have been split apart by different tokenizers, and use this representation to derive an approximation of the gradient of one model's outputs with respect to another model's embedding space. We show the efficacy of our approach via multi-objective optimization over vision-language and causal language models for image captioning and sentiment-based image captioning.

Figures (3)

• Figure 1: An $\mathbb{R}^{9 \times d \times 2}$ tensor vocabulary over words: "the quick brown fox jumps over the lazy dog". Each plain-text word represents its corresponding $\mathbb{R}^{d}$ embedding, and each $\textbf{\O}$ is a $0$ vector. We approximate the gradient for a mapping from model $\mathcal{M}_{i}$'s embeddings to $\mathcal{M}_{j}$'s embeddings by computing the t-product $\tilde{V}_{i}^{+} * \tilde{V}_{j}$, where $\tilde{V}_{i}^{+}$.
• Figure 2: The FUSE adapter connecting two transformer models for parallel inference. Inputs from Model 1 flow through the adapter by converting to text, retokenizing with Model 2's tokenizer, and embedding into Model 2's input space. The backward pass receives the gradient from Model 2, and multiplies it by the precomputed $\tilde{V}_{1}^{+} * \tilde{V}_{2}$.
• Figure 3: Example Captions that using a FUSE Adapter to minimize the sum of GPT2-Medium, CLIP-VIT-B/32, and a Bert-based Sentiment Classifier via AutoDAN zhu2023autodan. This combination of models controls through synonyms that indicate tone or through creating additional context for each image to denote tone. Note that AutoDAN does not have a clear stopping condition, a caption may stop in the middle of a sentence.