Language Model Maps for Prompt-Response Distributions via Log-Likelihood Vectors

Abstract

We propose a method that represents language models by log-likelihood vectors over prompt-response pairs and constructs model maps for comparing their conditional distributions. In this space, distances between models approximate the KL divergence between the corresponding conditional distributions. Experiments on a large collection of publicly available language models show that the maps capture meaningful global structure, including relationships to model attributes and task performance. The method also captures systematic shifts induced by prompt modifications and their approximate additive compositionality, suggesting a way to analyze and predict the effects of composite prompt operations. We further introduce pointwise mutual information (PMI) vectors to reduce the influence of unconditional distributions; in some cases, PMI-based model maps better reflect training-data-related differences. Overall, the framework supports the analysis of input-dependent model behavior.

Figures (5)

• Figure 1: Map of $552$ language models. Each model is represented by a conditional log-likelihood vector over prompt-response pairs from the Tulu3 text set and visualized using t-SNE. The color of each point indicates model performance, based on the average score on Open LLM Leaderboard v2. See Section \ref{['sec:conditional-model-map']} for details.
• Figure 2: Visualization of "prompt shift" on the model map constructed from the Tulu3 text set. For each model, we compute conditional log-likelihood vectors for the original prompt $x$ (base), the prompt with an added CoT phrase $T_{\mathrm{cot}}(x)$ (cot), the repeated prompt $T_{\mathrm{rep}}(x)$ (repeat), and the prompt with both transformations applied $T_{\mathrm{rep}+\mathrm{cot}}(x)$ (repeat+cot). These vectors are projected onto the linear subspace spanned by the mean shift vectors for $x \to T_{\mathrm{cot}}(x)$ and $x \to T_{\mathrm{rep}}(x)$, and visualized by PCA within that subspace. Gray points indicate the base positions, with model names omitted. Systematic shifts induced by prompt transformations and their approximate additive compositionality are clearly observed.
• Figure 3: Illustration of the difference between the prompt-response distribution setting considered in this paper and the unconditional distribution setting studied by modelmap2025. In both settings, log-likelihoods are computed for the response $y$. In our method, however, we compute $\log p(y | x)$ for each model, conditioning on the input prompt $x$.
• Figure 4: Comparison of the PMI model map and the conditional model map. Maps are constructed from PMI vectors (left) and conditional log-likelihood vectors (right) on Tulu3 and Infinity-Instruct. Colors indicate the mean PMI $\overline{\Delta \ell}_i$ in the PMI maps and the mean conditional log-likelihood $\bar{\ell}_i$ in the conditional maps. Models trained on Tulu3 or Infinity-Instruct are marked with stars, and their corresponding base models are marked with circles. The figure qualitatively illustrates that the PMI model map tends to reflect more clearly the correspondence between the training data and the text set used for visualization.
• Figure 5: Empirical comparisons of model distances. Left: comparison with a Monte Carlo estimate of KL divergence. Right: comparison with an embedding-based semantic distance.