Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition

Carolin Holtermann, Markus Frohmann, Navid Rekabsaz, Anne Lauscher

TL;DR

This work addresses how to effectively compose zero-shot knowledge from pretrained language models by introducing a unified framework that separates adapter selection, weighting, and final combination. Through a large-scale benchmarking study, it compares two combination methods (averaging vs. ensembling) and five scoring strategies (including uniform, SentSim, TF--IDF, domain prior, and entropy) across 21 training and 10 evaluation domains using GPT-2 and DeBERTa adapters. The findings show that ensembling typically outperforms simple averaging, corpus-based weighting strategies often outperform model-based ones, and increasing the number of adapters yields diminishing returns, while simple approaches can be surprisingly effective. A meta-regression analysis demonstrates that adapter-composition performance can be partially predicted from experimental metadata, offering a practical path to reducing search costs and guiding future research; the authors also release code for reproducibility.

Abstract

The knowledge encapsulated in a model is the core factor determining its final performance on downstream tasks. Much research in NLP has focused on efficient methods for storing and adapting different types of knowledge, e.g., in dedicated modularized structures, and on how to effectively combine these, e.g., by learning additional parameters. However, given the many possible options, a thorough understanding of the mechanisms involved in these compositions is missing, and hence it remains unclear which strategies to utilize. To address this research gap, we propose a novel framework for zero-shot module composition, which encompasses existing and some novel variations for selecting, weighting, and combining parameter modules under a single unified notion. Focusing on the scenario of domain knowledge and adapter layers, our framework provides a systematic unification of concepts, allowing us to conduct the first comprehensive benchmarking study of various zero-shot knowledge composition strategies. In particular, we test two module combination methods and five selection and weighting strategies for their effectiveness and efficiency in an extensive experimental setup. Our results highlight the efficacy of ensembling but also hint at the power of simple though often-ignored weighting methods. Further in-depth analyses allow us to understand the role of weighting vs. top-k selection, and show that, to a certain extent, the performance of adapter composition can even be predicted.

What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition

TL;DR

This work addresses how to effectively compose zero-shot knowledge from pretrained language models by introducing a unified framework that separates adapter selection, weighting, and final combination. Through a large-scale benchmarking study, it compares two combination methods (averaging vs. ensembling) and five scoring strategies (including uniform, SentSim, TF--IDF, domain prior, and entropy) across 21 training and 10 evaluation domains using GPT-2 and DeBERTa adapters. The findings show that ensembling typically outperforms simple averaging, corpus-based weighting strategies often outperform model-based ones, and increasing the number of adapters yields diminishing returns, while simple approaches can be surprisingly effective. A meta-regression analysis demonstrates that adapter-composition performance can be partially predicted from experimental metadata, offering a practical path to reducing search costs and guiding future research; the authors also release code for reproducibility.

Abstract

The knowledge encapsulated in a model is the core factor determining its final performance on downstream tasks. Much research in NLP has focused on efficient methods for storing and adapting different types of knowledge, e.g., in dedicated modularized structures, and on how to effectively combine these, e.g., by learning additional parameters. However, given the many possible options, a thorough understanding of the mechanisms involved in these compositions is missing, and hence it remains unclear which strategies to utilize. To address this research gap, we propose a novel framework for zero-shot module composition, which encompasses existing and some novel variations for selecting, weighting, and combining parameter modules under a single unified notion. Focusing on the scenario of domain knowledge and adapter layers, our framework provides a systematic unification of concepts, allowing us to conduct the first comprehensive benchmarking study of various zero-shot knowledge composition strategies. In particular, we test two module combination methods and five selection and weighting strategies for their effectiveness and efficiency in an extensive experimental setup. Our results highlight the efficacy of ensembling but also hint at the power of simple though often-ignored weighting methods. Further in-depth analyses allow us to understand the role of weighting vs. top-k selection, and show that, to a certain extent, the performance of adapter composition can even be predicted.
Paper Structure (39 sections, 7 equations, 14 figures, 5 tables)

This paper contains 39 sections, 7 equations, 14 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. A Unified Composition Framework
  4. Scoring Strategy
  5. Uniform.
  6. Semantic Sentence Similarity.
  7. TF--IDF.
  8. Domain Prior.
  9. Entropy.
  10. Combination Method
  11. Parameter Averaging.
  12. Ensembling.
  13. Benchmarking Composition Strategies
  14. Overall Experimental Setup
  15. Data.
  16. ...and 24 more sections

Figures (14)

  • Figure 1: Our unified framework for on-demand module composition consisting of three steps: selection, weighting, and final combination. We show the example of zero-shot domain adaptation with adapter layers.
  • Figure 2: Comparison between Parameter Averaging (solid lines) and Ensembling (dashed lines) over different numbers of top-$k$ adapters. We show the mean perplexity results for (a) gpt2-base, and (b) deberta-base for each of our scoring strategies (SentSim, tf--idf, entropy, prior) averaged across four runs.
  • Figure 3: Adapter weights for all training domains and scoring strategies when using all trained adapters. The light grey shade indicates the uniform weighting.
  • Figure 4: Comparison between weighting adapters based on their similarity (blue) and assigning them uniform weights (red). We show the mean perplexity results for (a) deberta-base, and (b) gpt2-base and when using corpus-based scoring strategies (tf--idf, SentSim) averaged over four runs and both combination strategies.
  • Figure 5: The different scoring and combination strategies with regards to their efficiency. We show the results for gpt2-base for Parameter Averaging (solid lines) and Ensembling (dashed lines) paired with each of our four scoring strategies and averaged across four runs.
  • ...and 9 more figures