Optimal Clustering from Noisy Binary Feedback

TL;DR

This work develops an adaptive algorithm that is inspired by the derivation of the information-theoretical error lower bounds, and in turn allocates the budget in an efficient way, and develops an adaptive algorithm that learns to select items hard to cluster and relevant questions more often.

Abstract

We study the problem of clustering a set of items from binary user feedback. Such a problem arises in crowdsourcing platforms solving large-scale labeling tasks with minimal effort put on the users. For example, in some of the recent reCAPTCHA systems, users clicks (binary answers) can be used to efficiently label images. In our inference problem, items are grouped into initially unknown non-overlapping clusters. To recover these clusters, the learner sequentially presents to users a finite list of items together with a question with a binary answer selected from a fixed finite set. For each of these items, the user provides a noisy answer whose expectation is determined by the item cluster and the question and by an item-specific parameter characterizing the {\it hardness} of classifying the item. The objective is to devise an algorithm with a minimal cluster recovery error rate. We derive problem-specific information-theoretical lower bounds on the error rate satisfied by any algorithm, for both uniform and adaptive (list, question) selection strategies. For uniform selection, we present a simple algorithm built upon the K-means algorithm and whose performance almost matches the fundamental limits. For adaptive selection, we develop an adaptive algorithm that is inspired by the derivation of the information-theoretical error lower bounds, and in turn allocates the budget in an efficient way. The algorithm learns to select items hard to cluster and relevant questions more often. We compare the performance of our algorithms with or without the adaptive selection strategy numerically and illustrate the gain achieved by being adaptive.

Figures (6)

• Figure 1: Images for Example 1.
• Figure 2: Model 1. (top) Global error rate vs. number of users. (bottom) Error rate for the 20% most difficult items vs. number of users. One standard deviations are shown using shaded areas.
• Figure 3: Model 1. The budget allocation under the adaptive algorithm vs. number of users. Items and questions are grouped into 4 categories, e.g. $(0-20\%, \ell=1,2)$ is the category regrouping the 20% most difficult items and questions $\ell=1,2$. One standard deviations are shown using shaded areas.
• Figure 4: Model 2. Global error rate vs. number of users. One standard deviations are shown using shaded areas.
• Figure 5: Model 3. Global error rate vs. number of users. One standard deviations are shown using shaded areas.

Theorems & Definitions (15)

• Example 1
• Theorem 1
• Proposition 1
• Corollary 1
• proof : Proof of Theorem \ref{['thm:lower-random']}
• Lemma 1
• Theorem 2
• proof : Proof of Theorem \ref{['thm:lower-adaptive']}
• Theorem 3
• proof : Proof of Theorem \ref{['thm:misclassification_after_phase_1']}