Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solidago: A Modular Collaborative Scoring Pipeline

Lê Nguyên Hoang, Romain Beylerian, Bérangère Colbois, Julien Fageot, Louis Faucon, Aidan Jungo, Alain Le Noac'h, Adrien Matissart, Oscar Villemaud

TL;DR

This paper presents Solidago, an end-to-end modular pipeline to allow any community of users to collaboratively score any number of entities, and proposes a six-module decomposition that uses pretrust and peer-to-peer vouches to assign trust scores to users.

Abstract

This paper presents Solidago, an end-to-end modular pipeline to allow any community of users to collaboratively score any number of entities. Solidago proposes a six-module decomposition. First, it uses pretrust and peer-to-peer vouches to assign trust scores to users. Second, based on participation, trust scores are turned into voting rights per user per entity. Third, for each user, a preference model is learned from the user's evaluation data. Fourth, users' models are put on a similar scale. Fifth, these models are securely aggregated. Sixth, models are post-processed to yield human-readable global scores. We also propose default implementations of the six modules, including a novel trust propagation algorithm, and adaptations of state-of-the-art scaling and aggregation solutions. Our pipeline has been successfully deployed on the open-source platform tournesol.app. We thereby lay an appealing foundation for the collaborative, effective, scalable, fair, interpretable and secure scoring of any set of entities.

Solidago: A Modular Collaborative Scoring Pipeline

TL;DR

This paper presents Solidago, an end-to-end modular pipeline to allow any community of users to collaboratively score any number of entities, and proposes a six-module decomposition that uses pretrust and peer-to-peer vouches to assign trust scores to users.

Abstract

This paper presents Solidago, an end-to-end modular pipeline to allow any community of users to collaboratively score any number of entities. Solidago proposes a six-module decomposition. First, it uses pretrust and peer-to-peer vouches to assign trust scores to users. Second, based on participation, trust scores are turned into voting rights per user per entity. Third, for each user, a preference model is learned from the user's evaluation data. Fourth, users' models are put on a similar scale. Fifth, these models are securely aggregated. Sixth, models are post-processed to yield human-readable global scores. We also propose default implementations of the six modules, including a novel trust propagation algorithm, and adaptations of state-of-the-art scaling and aggregation solutions. Our pipeline has been successfully deployed on the open-source platform tournesol.app. We thereby lay an appealing foundation for the collaborative, effective, scalable, fair, interpretable and secure scoring of any set of entities.
Paper Structure (90 sections, 18 theorems, 52 equations, 10 figures)

This paper contains 90 sections, 18 theorems, 52 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related work.
  4. Paper structure.
  5. Problem formulation and data
  6. Pretrust.
  7. Vouch data.
  8. Comparisons.
  9. Privacy
  10. Problem formulation.
  11. Trust propagation
  12. Sink vouch
  13. LipschiTrust
  14. Probabilistic intuition.
  15. Defining LipschiTrust.
  16. ...and 75 more sections

Key Result

Proposition 3.1

There is a unique vector $\textsc{trust}_{}$ such that $\left\lVert{\textsc{tr}^{t}_{} - \textsc{trust}_{}}\right\rVert_{1} \leq U \beta^t$ for all $t$. Moreover, this vector verifies $\textsc{trust}_{} = \min \left\lbrace \textsc{trust}^{pre}_{} + \beta V^T \textsc{trust}_{}, 1 \right\rbrace$.

Figures (10)

  • Figure 1: This figure describes the Solidago pipeline (slightly simplified). Green boxes correspond to public data, while black boxes are kept hidden. The purple boxes contain both public and private data. The pipeline is composed of 6 steps, namely (1) trust propagation, (2) voting rights assignment, (3) preference learning, (4) model scaling, (5) model aggregation and (6) post-process.
  • Figure 2: Tournesol's users are asked to select any two videos and to report which they would recommend more widely.
  • Figure 3: Correlation between the ground truth and the learned global scores, as a function of the fraction of trustworthy users (left), and as a function of engagement bias (right).
  • Figure 4: Distribution of displayed user scores $\theta^{\bf display}_{u e}$ (top graph) and global $\rho^{\bf display}_{e}$ (bottom graph). Hyperparameters were selected so that users' scores have roughly a similar distribution, once they were compared at least 3 times, and so that the global scores of highly compared videos are well spread. Hyperparameter selection is further discussed in Appendix \ref{['sec:section']}.
  • Figure 5: Impact of the prior weight $\alpha_{prior}^{user}$ on the distributions of squashed individual and global scores
  • ...and 5 more figures

Theorems & Definitions (36)

  • Proposition 3.1
  • proof : Proof sketch
  • Theorem 3.2
  • proof : Proof sketch
  • Proposition 3.3
  • Proposition 3.4
  • Proposition 5.1: Theorem 2 in GBT2023
  • Theorem 7.1: Lipschitz resilience of QrQtl
  • proof : Proof sketch
  • Lemma A.1
  • ...and 26 more