U-aggregation: Unsupervised Aggregation of Multiple Learning Algorithms

TL;DR

U-aggregation tackles unsupervised aggregation of multiple pre-trained models in target populations without observed outcomes. It combines variance stabilization based on the Dyson equation with an Approximate Message Passing (AMP) algorithm to recover a sparse loading vector and the underlying risk vector, enabling robust aggregation under heteroskedastic noise and adversarial models. Theoretical results establish the consistency of variance stabilization and the high-dimensional limits of AMP estimators, including a BBP-type phase transition, while simulations and an AoU/PGS Catalog application demonstrate improved predictive coherence and model ranking without labels. This approach offers a practical tool for open-science contexts to leverage diverse pre-trained models across heterogeneous populations.

Abstract

Across various domains, the growing advocacy for open science and open-source machine learning has made an increasing number of models publicly available. These models allow practitioners to integrate them into their own contexts, reducing the need for extensive data labeling, training, and calibration. However, selecting the best model for a specific target population remains challenging due to issues like limited transferability, data heterogeneity, and the difficulty of obtaining true labels or outcomes in real-world settings. In this paper, we propose an unsupervised model aggregation method, U-aggregation, designed to integrate multiple pre-trained models for enhanced and robust performance in new populations. Unlike existing supervised model aggregation or super learner approaches, U-aggregation assumes no observed labels or outcomes in the target population. Our method addresses limitations in existing unsupervised model aggregation techniques by accommodating more realistic settings, including heteroskedasticity at both the model and individual levels, and the presence of adversarial models. Drawing on insights from random matrix theory, U-aggregation incorporates a variance stabilization step and an iterative sparse signal recovery process. These steps improve the estimation of individuals' true underlying risks in the target population and evaluate the relative performance of candidate models. We provide a theoretical investigation and systematic numerical experiments to elucidate the properties of U-aggregation. We demonstrate its potential real-world application by using U-aggregation to enhance genetic risk prediction of complex traits, leveraging publicly available models from the PGS Catalog.

Figures (7)

• Figure 1: Values of $\mu_t/(\lambda\sigma_{t+1})$ (Left) and $\bar{\mu}_t/(\sqrt{\alpha}\lambda\sigma_{t+1})$ (Right) of the AMP algorithm over iteration numbers $t$. The specifications of $\lambda, \alpha$, and $\omega$ are shown in the plot. $\nu_u^*$ is a mixture distribution, consisting of a uniform distribution over $(0, c)$ with probability $\omega$, and a point mass at $0$ with probability $1 - \omega$. We choose $c$ such that the second moment of $\nu_u^*$ is $1$. The two plots on the same row have the same parameter setting.
• Figure 2: Illustration of the asymptotic limits $\mu^*/(\lambda\sigma^*)$ (Green) and $\bar{\mu}^*/(\sqrt{\alpha}\lambda\sigma^*)$ (Blue) as functions of $\lambda$ (Upper Panel) and $\alpha$ (Lower Panel). In the upper panel, we choose $\alpha = 0.3$ and $\omega = 0.3$. In the lower panel, we choose $\lambda = 2$, and $\omega = 0.5$. In all settings, $\nu_u^*$ is a mixture distribution, consisting of a uniform distribution over $(0, c)$ with probability $\omega$, and a point mass at $0$ with probability $1 - \omega$. We choose $c$ such that the second moment of $\nu_u^*$ is $1$.
• Figure 3: Performance of the compared methods in estimating $\bold{v}$ across various simulation settings.
• Figure 4: Correlation between the assigned weights and the performance of each pre-trained model across various simulation settings.
• Figure 5: The ratios between consecutive singular values in four datasets from the AoU Study, each containing the pretrained PRSs for a specific trait. It suggests approximate rank-one structures in these matrices.

Theorems & Definitions (3)

• Theorem 3.1
• Definition 3.2
• Theorem 3.3