Task Priors: Enhancing Model Evaluation by Considering the Entire Space of Downstream Tasks

TL;DR

This work addresses the evaluation bottleneck in AI by proposing Task Priors, a probabilistic framework that treats downstream tasks as samples from a distribution over label graphs driven by a data kernel. By establishing an equivalence between absolute and relative losses and introducing a Gibbs prior on label graphs, the authors derive closed-form expressions for the expected downstream error and its variance, enabling task-agnostic evaluation without additional training. They also provide an efficient prefix-sampling algorithm to draw realistic tasks and validate that the resulting kernel-based statistics correlate with and predict linear-probe performance and align with curated benchmarks like MIEB. The approach offers a principled, scalable alternative to fixed benchmark suites, potentially accelerating SSL research by providing robust, distributional performance signals across the vast space of downstream tasks. In practice, Task Priors yield practical metrics for average performance, robustness, and worst-case considerations across tasks, with broad implications for evaluating and comparing representation learning methods.

Abstract

The grand goal of AI research, and particularly Self Supervised Learning (SSL), is to produce systems that can successfully solve any possible task. In contrast, current evaluation methods available to AI researchers typically rely on a fixed collection of hand-picked downstream benchmarks. Hence, a large amount of effort is put into designing and searching for large collection of evaluation tasks that can serve as a proxy of our grand goal. We argue that such a rigid evaluation protocol creates a silent bottleneck in AI research. To remedy that, we define a probabilistic space of downstream tasks obtained by adopting a distribution of tasks and by defining Task Priors. Under this view, one can evaluate a model's performance over the set of all possible downstream tasks. Our framework is the first to provide answers to key questions such as (i) what is the average performance of my model over all possible downstream tasks weighted by the probability to encounter each task? or (ii) what is the variance of my model's performance across all downstream tasks under the defined Task Priors? Beyond establishing a new standard for evaluation, we believe that Task Priors will accelerate the pace of research in SSL - where downstream task evaluation is the sole qualitative signal that researchers have access to.

Figures (8)

• Figure 1: Comparison of the naive way to evaluate a model, only on the specific choice of labels provided with the Imagenette Howard_Imagenette_2019 dataset (Left) with the probabilistic view of targets sampled from the Task Prior, giving us a distribution we can evaluate on (Right).
• Figure 2: Here we show an example of using Task Priors to evaluate the DinoV2 oquab2023dinov2 family of models, with respect the the Task Prior kernel, dinov2-giant. We show the expectation and variance of $\mathop{\mathrm{Tr}}\nolimits({\mathbf{K}}{\mathbf{G}})$ (Left), as well as the distribution of performance of linear probes on labelings sampled from the Task Prior (Right).
• Figure 3: We plot the expectation and variance of $\operatorname{Tr}(GM)$, where $M$ is the centered cosine similarity kernel matrix for each models features generated from mini-imagenet imagenet15russakovsky, where the expectation is taken against $\mu_K$. Please see the appendix for more information and ablation on temperature and choice of prior kernel.
• Figure 4: Correlation between the mean and variance of $\operatorname{Tr}(GK)$, and the accuracy of linear probes sampled by the same Task Prior. We observe a Spearman correlation of $0.68$ (top) and $0.76$ (bottom).
• Figure 5: For each of the $26$ models, we estimate the task-prior expectation and variance and compare them to that model’s mean accuracy over a hand-curated set of $22$ MIEB classification tasks xiao2025miebmassiveimageembedding. We observe a Spearman correlation of $0.82$ (Left) and $0.74$ (Right), showcasing how Task Priors can accurately predict model performance on a distribution of tasks.

Theorems & Definitions (10)

• Definition 2.1: Absolute objective
• Definition 2.2: Relative objective
• Theorem 2.3
• Definition 2.4
• Lemma 2.5
• Theorem 2.6
• Proposition 2.7
• proof
• proof
• proof