Liquid Democracy for Low-Cost Ensemble Pruning

TL;DR

This paper links ensemble pruning with liquid democracy to drastically cut training costs for large classifier ensembles without sacrificing accuracy. By using incremental delegation-based pruning, classifiers transfer weight to better performers and are pruned over successive data increments, yielding a smaller, effective ensemble. Among the delegation schemes, Proportional Weighted strikes the strongest accuracy-cost balance, often outperforming Adaboost variants on several datasets while reducing compute. The approach also provides a natural framework for scalable, online, or out-of-core learning and opens avenues for theory-backed analyses of delegation quality and alternative scheduling strategies.

Abstract

We argue that there is a strong connection between ensemble learning and a delegative voting paradigm -- liquid democracy -- that can be leveraged to reduce ensemble training costs. We present an incremental training procedure that identifies and removes redundant classifiers from an ensemble via delegation mechanisms inspired by liquid democracy. Through both analysis and extensive experiments we show that this process greatly reduces the computational cost of training compared to training a full ensemble. By carefully selecting the underlying delegation mechanism, weight centralization in the classifier population is avoided, leading to higher accuracy than some boosting methods. Furthermore, this work serves as an exemplar of how frameworks from computational social choice literature can be applied to problems in nontraditional domains.

Figures (20)

• Figure 1: A possible outcome of incremental pruning on a small ensemble over 3 time steps. At each increment, each remaining classifier updates its average accuracy according to its performance on the most recent training data. At $t = 1$, each classifier is weighted equally and trained on one increment of data to generate an estimate of their accuracy. $v_1$ has the lowest estimated accuracy while $v_4$ has the highest. At $t = 2$, the weight of $v_1$, the "weakest" classifier during $t = 1$ is transferred to the "strongest" classifier, $v_4$ and the 4 classifiers remaining with non-zero weight train on another increment of data and estimates of their accuracy are refined. Finally, at $t = 3$, as $v_3$ is the classifier with the lowest accuracy estimate its weight is transferred to the more accurate $v_1$. After two increments of pruning, future training of the ensemble will use only 60% of the original training cost. Note: The actual choice of which classifiers are removed and where their weight is transferred depends upon parameters described in \ref{['sec:delegation_mechanisms']}.
• Figure 2: Two possible states when an ensemble composed of 5 classifiers predicts the classes of 5 examples. A cell shows whether a particular voter (rows) is correct or incorrect in classifying each example (columns). (a) All voters are pivotal and only examples $m_4$ and $m_5$ are pivotal. If any non-pivotal examples $m_1$, $m_2$, or $m_3$ were removed all voters in (a) would remain pivotal. (b) All voters are pivotal and all examples are pivotal.
• Figure 3: The lower bound on delegation cost as $n^\text{final}$ and delegation rate are varied.
• Figure 4: Test accuracy of fully trained ensemble across delegation methods as parameters affecting accuracy are varied. Results displayed are from the spambase dataset. Random delegations are omitted as they perform significantly worse than the displayed delegation mechanisms; Direct delegations are omitted as increment size and delegate rate do not affect Direct ensemble performance.
• Figure 5: (left) Test accuracy during training on spambase dataset, averaged over 500 trials. (right) Minimum majority size during training on the spambase dataset.

Theorems & Definitions (15)

• Definition 3.1: Delegation Mechanism
• Definition 3.2
• Definition 3.3
• Definition 3.4
• Definition 3.5
• Definition 3.6
• Definition 3.7
• Definition 3.8
• Definition 3.9
• Definition 3.10