Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stability of Weighted Majority Voting under Estimated Weights

Shaojie Bai, Dongxia Wang, Tim Muller, Peng Cheng, Jiming Chen

TL;DR

This work analyzes Weighted Majority Voting under estimated source trust, introducing two stability notions: Stability of Correctness (SoC) and Stability of Optimality (SoO). It proves SoC holds absolutely when trust estimates are unbiased, meaning the decision-maker’s perceived accuracy matches actual accuracy regardless of the trust distribution; however, SoO does not hold in general, though its degradation is bounded and depends on the distribution of trustworthiness. The authors develop a formal framework, including parameter sensitivity analyses and a stability analysis that derives bounds on SoO and demonstrates SoC via Monte Carlo simulations. The results support the practical reliability of WMV with unbiased trust, and quantify how estimation uncertainty affects decision quality, with implications for crowdsourcing and ensemble learning where trust is learned rather than known.

Abstract

Weighted Majority Voting (WMV) is a well-known optimal decision rule for collective decision making, given the probability of sources to provide accurate information (trustworthiness). However, in reality, the trustworthiness is not a known quantity to the decision maker - they have to rely on an estimate called trust. A (machine learning) algorithm that computes trust is called unbiased when it has the property that it does not systematically overestimate or underestimate the trustworthiness. To formally analyse the uncertainty to the decision process, we introduce and analyse two important properties of such unbiased trust values: stability of correctness and stability of optimality. Stability of correctness means that the decision accuracy that the decision maker believes they achieved is equal to the actual accuracy. We prove stability of correctness holds. Stability of optimality means that the decisions made based on trust, are equally good as they would have been if they were based on trustworthiness. Stability of optimality does not hold. We analyse the difference between the two, and bounds thereon. We also present an overview of how sensitive decision correctness is to changes in trust and trustworthiness.

Stability of Weighted Majority Voting under Estimated Weights

TL;DR

This work analyzes Weighted Majority Voting under estimated source trust, introducing two stability notions: Stability of Correctness (SoC) and Stability of Optimality (SoO). It proves SoC holds absolutely when trust estimates are unbiased, meaning the decision-maker’s perceived accuracy matches actual accuracy regardless of the trust distribution; however, SoO does not hold in general, though its degradation is bounded and depends on the distribution of trustworthiness. The authors develop a formal framework, including parameter sensitivity analyses and a stability analysis that derives bounds on SoO and demonstrates SoC via Monte Carlo simulations. The results support the practical reliability of WMV with unbiased trust, and quantify how estimation uncertainty affects decision quality, with implications for crowdsourcing and ensemble learning where trust is learned rather than known.

Abstract

Weighted Majority Voting (WMV) is a well-known optimal decision rule for collective decision making, given the probability of sources to provide accurate information (trustworthiness). However, in reality, the trustworthiness is not a known quantity to the decision maker - they have to rely on an estimate called trust. A (machine learning) algorithm that computes trust is called unbiased when it has the property that it does not systematically overestimate or underestimate the trustworthiness. To formally analyse the uncertainty to the decision process, we introduce and analyse two important properties of such unbiased trust values: stability of correctness and stability of optimality. Stability of correctness means that the decision accuracy that the decision maker believes they achieved is equal to the actual accuracy. We prove stability of correctness holds. Stability of optimality means that the decisions made based on trust, are equally good as they would have been if they were based on trustworthiness. Stability of optimality does not hold. We analyse the difference between the two, and bounds thereon. We also present an overview of how sensitive decision correctness is to changes in trust and trustworthiness.
Paper Structure (12 sections, 7 theorems, 10 equations, 7 figures)

This paper contains 12 sections, 7 theorems, 10 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Parameter Sensitivity
  5. Direct Sensitivity Analysis
  6. Trustworthiness Sensitivity Analysis
  7. Trust Sensitivity Analysis
  8. Stability
  9. Parameter Distributions
  10. Stability of Correctness
  11. Stability of Optimality
  12. Conclusion and Future Work

Key Result

Lemma 1

Let $f(p_i) = \omega(\bm{p},\bm{p})$, where $p_j$ is constant for $j \neq i$. The function $f(p_i)$ is a piecewise linear non-decreasing convex function.

Figures (7)

  • Figure 1: Sensitivity of $\mathcal{D}_{W}$ to $\bm{p}$ when $\bm{p} = \hat{\bm{p}}$
  • Figure 2: Accuracy of $\mathcal{D}_{W}$ with $m$ sources being identical
  • Figure 3: Sensitivity of $\bm{p}$ with $\hat{\bm{p}}$ fixed
  • Figure 4: Sensitivity of $\hat{\bm{p}}$ with $\bm{p}$ fixed
  • Figure 5: Effect of variance on Stability of Correctness
  • ...and 2 more figures

Theorems & Definitions (10)

  • Example 1
  • Definition 1: Weighted Majority Voting $\mathcal{D}_{W}$
  • Example 2
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Theorem 1
  • Lemma 4
  • Corollary 1
  • Theorem 2