Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised Ensemble Learning Through Deep Energy-based Models

Ariel Maymon, Yanir Buznah, Uri Shaham

TL;DR

Unsupervised ensemble learning aims to recover the true labels $Y$ from predictions $X=(X_1,\dots,X_d)$ without labeled data. The authors reformulate the Dawid–Skene model as a deep energy-based model built on an identifiable Fully Multinomial RBM (iRBM), and extend it with deep Multinomial layers (DEEM) to relax conditional independence. They prove identifiability and a bijection between the iRBM and the CI model, show that the posterior $p_\theta(Y|X)$ can be recovered, and train DEEM with a Deep Langevin Proposal, aligning outputs via the Hungarian algorithm. Across synthetic and real-world ensembles, DEEM achieves state-of-the-art performance, effectively detecting expert subnetworks and scaling to large class spaces such as $K=1000$ ImageNet classes. This work provides a practical unsupervised framework for fusing diverse predictions in data-scarce or privacy-sensitive contexts.

Abstract

Unsupervised ensemble learning emerged to address the challenge of combining multiple learners' predictions without access to ground truth labels or additional data. This paradigm is crucial in scenarios where evaluating individual classifier performance or understanding their strengths is challenging due to limited information. We propose a novel deep energy-based method for constructing an accurate meta-learner using only the predictions of individual learners, potentially capable of capturing complex dependence structures between them. Our approach requires no labeled data, learner features, or problem-specific information, and has theoretical guarantees for when learners are conditionally independent. We demonstrate superior performance across diverse ensemble scenarios, including challenging mixture of experts settings. Our experiments span standard ensemble datasets and curated datasets designed to test how the model fuses expertise from multiple sources. These results highlight the potential of unsupervised ensemble learning to harness collective intelligence, especially in data-scarce or privacy-sensitive environments.

Unsupervised Ensemble Learning Through Deep Energy-based Models

TL;DR

Unsupervised ensemble learning aims to recover the true labels from predictions without labeled data. The authors reformulate the Dawid–Skene model as a deep energy-based model built on an identifiable Fully Multinomial RBM (iRBM), and extend it with deep Multinomial layers (DEEM) to relax conditional independence. They prove identifiability and a bijection between the iRBM and the CI model, show that the posterior can be recovered, and train DEEM with a Deep Langevin Proposal, aligning outputs via the Hungarian algorithm. Across synthetic and real-world ensembles, DEEM achieves state-of-the-art performance, effectively detecting expert subnetworks and scaling to large class spaces such as ImageNet classes. This work provides a practical unsupervised framework for fusing diverse predictions in data-scarce or privacy-sensitive contexts.

Abstract

Unsupervised ensemble learning emerged to address the challenge of combining multiple learners' predictions without access to ground truth labels or additional data. This paradigm is crucial in scenarios where evaluating individual classifier performance or understanding their strengths is challenging due to limited information. We propose a novel deep energy-based method for constructing an accurate meta-learner using only the predictions of individual learners, potentially capable of capturing complex dependence structures between them. Our approach requires no labeled data, learner features, or problem-specific information, and has theoretical guarantees for when learners are conditionally independent. We demonstrate superior performance across diverse ensemble scenarios, including challenging mixture of experts settings. Our experiments span standard ensemble datasets and curated datasets designed to test how the model fuses expertise from multiple sources. These results highlight the potential of unsupervised ensemble learning to harness collective intelligence, especially in data-scarce or privacy-sensitive environments.
Paper Structure (52 sections, 3 theorems, 36 equations, 13 figures, 10 tables, 2 algorithms)

This paper contains 52 sections, 3 theorems, 36 equations, 13 figures, 10 tables, 2 algorithms.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Ensemble Using Additional Information
  4. Ensemble Using Learners Predictions
  5. Relation to Programmatic Weak Supervision
  6. BACKGROUND
  7. Unsupervised Ensemble Learning
  8. The Conditional Independence Model
  9. Fully Multinomial RBM
  10. PROPOSED APPROACH
  11. The iRBM as a DS Model
  12. iRBM for Ensemble Learning
  13. Deep Energy Ensemble Models
  14. The Multinomial Layer
  15. Training and Inference Details of DEEM
  16. ...and 37 more sections

Key Result

Lemma 1

The joint probability $p_\lambda(V=x,H=y)$ of an iRBM with the parameters $\lambda = (W,a,b)$ with $d_v=d$ and $d_h=1$, is equivalent to the joint probability $p_\theta(X=x,Y=y)$ of the conditional independence model with the parameters $\theta = (\{\psi_{ilm}\},\{\pi_t\})$ given by: where $\sigma$ is the softmax function, $z_{ilm} = (a_i^l + w_i^{lm})$, and $\mathbf{z} = (z_{i1m},\dots,z_{iKm})

Figures (13)

  • Figure 1: Fully-Multinomial RBM, with $d_v,d_h$ multinomial units, each with size $K$. The weights tensor $W$ connects from every node in one layer to all nodes in the other, and vice versa.
  • Figure 2: The DEEM model. The sample $x_i$ goes through the deep network, to acquire the visible input $v_i$ to the iRBM layer, which then forwards it to get $h_i$. In training, $v_i,h_i$ are used to compute the gradient which then propagates backwards. When inferencing on $x_i$, the prediction $h_i$ is mapped to the correct class and returned.
  • Figure 3: Recovery graph (left) and a weight correlation heatmap (right). In the recovery graph, each circle is the parameter value from the DS model $(\psi_{ilm} , \pi_t)$ (X-axis), and its corresponding iRBM parameter value (Y-axis), using the map outlined in Lemma \ref{['lem:ci-rbm']}. The closer to the dotted identity line, the better the recovery (as it means they have the same value). It can be seen that the iRBM recovers the DS model original parameters correctly. The heatmap shows each classifier's correlation with the iRBM final prediction, which shows the iRBM was able to tell the classifiers that benefited the prediction, and rule out the rest (which were random guesses).
  • Figure 4: The conditional mutual information matrices of a trained DEEM with 2 multinomial layers. Starting from the input layer in the leftmost column, it can be seen how the mutual information is gradually reduced as we progress through the multinomial network, until we get disentangled features as input for the iRBM component. This is a typical plot from one particular true label class subset of the data.
  • Figure 5: Learner importance chart on MnistE-568. Red bars are the enhanced oracle learners. DEEM strongly favors the specialized on the expert, where they are relevant (left panel), while preserving their respective contribution on the rest of the data (right panel).
  • ...and 8 more figures

Theorems & Definitions (10)

  • Remark 1
  • Definition 1
  • Lemma 1
  • Remark 2
  • Corollary 1
  • Corollary 2
  • Remark 3
  • Remark 4
  • proof
  • proof