LOCUS: Low-Dimensional Model Embeddings for Efficient Model Exploration, Comparison, and Selection

TL;DR

LOCUS addresses the challenge of managing a large, evolving pool of LLMs by learning a fixed-dimensional embedding for each model that summarizes capability across queries. It introduces an attention-based encoder F_ heta that tokenizes and processes query evaluations, augmented by a correctness predictor G_psi to predict per-query correctness, enabling training-free onboarding of new models. The embedding dimension is $d=128$ with two latent-bottleneck blocks ($L=2$, $r=64$) that reduce attention complexity from $O(n_m^2)$ to $O(n_m r)$ while preserving information flow, and the space geometry correlates with model similarity, enabling clustering, nearest-neighbor proxies, and portfolio selection. Empirically, LOCUS achieves up to $4.8\times$ better sample efficiency than baselines, maintains robust routing and correctness predictions, and supports practical utilities such as resilient routing and efficient model portfolios, with embeddings remaining stable under varying query sets and enabling secure fingerprinting insights.

Abstract

The rapidly growing ecosystem of Large Language Models (LLMs) makes it increasingly challenging to manage and utilize the vast and dynamic pool of models effectively. We propose LOCUS, a method that produces low-dimensional vector embeddings that compactly represent a language model's capabilities across queries. LOCUS is an attention-based approach that generates embeddings by a deterministic forward pass over query encodings and evaluation scores via an encoder model, enabling seamless incorporation of new models to the pool and refinement of existing model embeddings without having to perform any retraining. We additionally train a correctness predictor that uses model embeddings and query encodings to achieve state-of-the-art routing accuracy on unseen queries. Experiments show that LOCUS needs up to 4.8x fewer query evaluation samples than baselines to produce informative and robust embeddings. Moreover, the learned embedding space is geometrically meaningful: proximity reflects model similarity, enabling a range of downstream applications including model comparison and clustering, model portfolio selection, and resilient proxies of unavailable models.

Figures (19)

• Figure 1: Routing accuracy vs number of training samples.LOCUS uses $\mathbf{2.3-4.8}\times$ fewer training samples (number of queries on which each model is evaluated) as compared to baselines, while maintaining high routing accuracy performance.
• Figure 2: Model Embedding Generator $F_\theta$
• Figure 3: Correctness Predictor $G_\psi$
• Figure 5: New LLM onboarding. Correctness prediction accuracy on $16$ held-out models: with varying number of query evaluations for embedding held-out models and varying number of models present in training data (top row indicates $(F_\theta, G_\psi)$ trained on the full set of all language models). Only a few queries ($\approx$$128$) are required to generate informative embeddings for unseen models, and there is a negligible ($< 1\%$) accuracy gap between using an encoder trained on the full (top row) versus a partial set of models.
• Figure 6: Robustness to Varying Query Evaluations. Accuracy for correctness prediction (top) and routing (bottom) as a function of (left) the number of evaluations used to construct embeddings and (right) the overlap fraction with a reference evaluation set. Performance saturates with as few as $128\text{-}256$ evaluations, and remains stable across overlap fractions and query subsampling, indicating robustness to evaluation queries.