UniAdapt: A Universal Adapter for Knowledge Calibration

TL;DR

UniAdapt, a universal adapter for knowledge calibration, is introduced, a universal adapter for knowledge calibration designed for seamless plug-and-play integration in large Language Models.

Abstract

Large Language Models (LLMs) require frequent updates to correct errors and keep pace with continuously evolving knowledge in a timely and effective manner. Recent research in it model editing has highlighted the challenges in balancing generalization and locality, especially in the context of lifelong model editing. We discover that inserting knowledge directly into the model often causes conflicts and potentially disrupts other unrelated pre-trained knowledge. To address this problem, we introduce UniAdapt, a universal adapter for knowledge calibration. Inspired by the Mixture of Experts architecture and Retrieval-Augmented Generation, UniAdapt is designed with a vector-assisted router that is responsible for routing inputs to appropriate experts. The router maintains a vector store, including multiple shards, to construct routing vectors based on semantic similarity search results. UniAdapt is fully model-agnostic and designed for seamless plug-and-play integration. Experimental results show that UniAdapt outperforms existing lifelong model editors and achieves exceptional results in most metrics.

Figures (6)

• Figure 1: The architecture of UniAdapt inspired by MoE architecture. UniAdapt contains a router and multiple parallel feed-forward layers (a.k.a experts), denoted as $FFN_1, FFN_2, \cdots, FFN_k$. The router maintains a vector store containing multiple shards labeled $S_1, S_2, \cdots, S_k$. The matching colors of shards and experts indicate that each expert may hold knowledge relevant to queries associated with the shard. In the inference phase, the router computes a routing vector to selectively choose appropriate $FFNs$, ensuring precise calibration of the original MLP's output (more details in \ref{['sect:knowledge_router']}).
• Figure 2: An example of the router's functionality, similar to a retriever in RAG. Instead of retrieving related documents, the router computes decision vectors based on the similarity scores. The similarity scores [1.0, 0.4, 0.3] indicate that there are three shards. The first shard has the highest similarity score thus the answer will be stored in expert 1 (also known as FFN1).
• Figure 3: Effect of Target Layer
• Figure 4: Effect of Expert Number
• Figure 5: Effect of $\epsilon$