Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Likelihood Variance as Text Importance for Resampling Texts to Map Language Models

Momose Oyama, Ryo Kishino, Hiroaki Yamagiwa, Hidetoshi Shimodaira

TL;DR

This work tackles the high computational cost of constructing a language-model map from log-likelihood vectors by introducing resampling strategies that prioritize texts with high variance across models. The proposed Length Squared (LS) and KL sampling methods assign text-selection probabilities to minimize distance-estimation error, enabling accurate KL-based model maps with roughly half the texts required by uniform sampling. Empirically, LS/KL sampling matches or closely approaches the baseline in estimating model distances, yields stable maps, and efficiently accommodates new models. Importantly, downstream performance predictions remain robust as the number of unique texts grows, indicating practical applicability for scalable model comparison and integration in dynamic model ecosystems.

Abstract

We address the computational cost of constructing a model map, which embeds diverse language models into a common space for comparison via KL divergence. The map relies on log-likelihoods over a large text set, making the cost proportional to the number of texts. To reduce this cost, we propose a resampling method that selects important texts with weights proportional to the variance of log-likelihoods across models for each text. Our method significantly reduces the number of required texts while preserving the accuracy of KL divergence estimates. Experiments show that it achieves comparable performance to uniform sampling with about half as many texts, and also facilitates efficient incorporation of new models into an existing map. These results enable scalable and efficient construction of language model maps.

Likelihood Variance as Text Importance for Resampling Texts to Map Language Models

TL;DR

This work tackles the high computational cost of constructing a language-model map from log-likelihood vectors by introducing resampling strategies that prioritize texts with high variance across models. The proposed Length Squared (LS) and KL sampling methods assign text-selection probabilities to minimize distance-estimation error, enabling accurate KL-based model maps with roughly half the texts required by uniform sampling. Empirically, LS/KL sampling matches or closely approaches the baseline in estimating model distances, yields stable maps, and efficiently accommodates new models. Importantly, downstream performance predictions remain robust as the number of unique texts grows, indicating practical applicability for scalable model comparison and integration in dynamic model ecosystems.

Abstract

We address the computational cost of constructing a model map, which embeds diverse language models into a common space for comparison via KL divergence. The map relies on log-likelihoods over a large text set, making the cost proportional to the number of texts. To reduce this cost, we propose a resampling method that selects important texts with weights proportional to the variance of log-likelihoods across models for each text. Our method significantly reduces the number of required texts while preserving the accuracy of KL divergence estimates. Experiments show that it achieves comparable performance to uniform sampling with about half as many texts, and also facilitates efficient incorporation of new models into an existing map. These results enable scalable and efficient construction of language model maps.

Paper Structure

This paper contains 42 sections, 3 theorems, 41 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Map of Language Models
  4. Length Squared Sampling
  5. Resampling Texts for Model Map
  6. Text Resampling Method
  7. LS Sampling.
  8. KL Sampling.
  9. Baseline: Uniform Sampling.
  10. Model Map with Resampled Texts
  11. Experiments
  12. Error of Resampled Estimates
  13. Resampling Error to Population ($\hat{\sigma}_{\mathrm{Method}, n}$).
  14. Baseline: Sampling Error ($\hat{\kappa}_m$).
  15. Results.
  16. ...and 27 more sections

Key Result

Lemma 1

For any $i, j \in \{1, \ldots, K\}$, it holds that

Figures (6)

  • Figure 1: The model map calculated with $D_N$ is visualized using t-SNE. The sampling error for each point was estimated using the bootstrap resampling method and shown as an ellipse. See Appendix \ref{['sec:app-model-map']} for details.
  • Figure 2: The number of unique texts $d$ for each resampling method and the estimation error against the population. 'Sampling Error' corresponds to the estimation error when $d$ texts are randomly sampled without replacement from the population $D^{\dagger}$. Here, the error is normalized by KL divergence for each model pair. For error evaluation without normalization, see Appendix \ref{['sec:absolute-error']}.
  • Figure 3: (a) Model maps based on LS ($n=2900, d = 2202$), uniform ($n=10000, d = 6320$), and uniform ($n=2500, d = 2209$) sampling. Each map shows the mean coordinates and their variability as ellipses across 100 trials. (b) Model maps with 120 new models added to existing 898 models. The left panel uses $d = 2210$ unique texts selected by LS sampling with $n = 2900$. The right panel uses all $N = 10{,}000$ texts.
  • Figure 4: Pearson's correlation $r$ between predicted and actual scores on the average of six downstream tasks, shown as a function of the number of unique texts $d$.
  • Figure 5: Result of error evaluation without normalization for each model pair. Other settings are the same as Fig. \ref{['fig:diff-ratio-distance']}.
  • ...and 1 more figures

Theorems & Definitions (6)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Proposition 1
  • proof