L2CU: Learning to Complement Unseen Users

TL;DR

L2CU tackles the challenge of generalizing learning-to-complement to unseen users by uncovering annotator-specific labeling patterns and training profile-aware cooperative models. It identifies representative annotator profiles from sparse, noisy multi-rater data, augments data per profile, and matches unseen users to a profile at test time to select a corresponding cooperative model. The approach yields consistent improvements over L2D and L2C baselines across image and text domains, validated on CIFAR-10N, CIFAR-10H, Fashion-MNIST-H, Chaoyang, and AgNews, and introduces the alteration-rate metric to quantify human-label impact. With a model-agnostic design and no requirement for ground-truth during training, L2CU offers a practical path to robust human-AI cooperation in real-world, multi-rater settings.

Abstract

Recent research highlights the potential of machine learning models to learn to complement (L2C) human strengths; however, generalizing this capability to unseen users remains a significant challenge. Existing L2C methods oversimplify interaction between human and AI by relying on a single, global user model that neglects individual user variability, leading to suboptimal cooperative performance. Addressing this, we introduce L2CU, a novel L2C framework for human-AI cooperative classification with unseen users. Given sparse and noisy user annotations, L2CU identifies representative annotator profiles capturing distinct labeling patterns. By matching unseen users to these profiles, L2CU leverages profile-specific models to complement the user and achieve superior joint accuracy. We evaluate L2CU on datasets (CIFAR-10N, CIFAR-10H, Fashion-MNIST-H, Chaoyang and AgNews), demonstrating its effectiveness as a model-agnostic solution for improving human-AI cooperative classification.

Figures (4)

• Figure 1: Paradigms of Human-AI Cooperation with seen users in blue and unseen users in orange. L2D defers decisions to humans, evolving to handle both seen and unseen users. L2C complements human strengths, and L2CU (Ours) advances L2C to complement unseen users.
• Figure 2: Three step L2CU framework. 1) During training, from a sparse multi-rater dataset, unique annotator profiles are identified $(1,...,K)$. Then, for each annotator profile, noisy label augmentation is performed and a AI cooperative model is trained. 2) During user profiling, a test user annotates a validation set and based on validation labels, a profile is matched by OVA SVM, entry condition is evaluated, and respective AI cooperative model is selected. 3) At inference, the test user is paired with the corresponding model from the selected profile for cooperative classification.
• Figure 3: Positive alterations made by the L2CU on CIFAR-10N, Fashion-MNIST-H and Chaoyang experiments (top to bottom).
• Figure 4: Estimated noise matrices for identified annotator profiles from CIFAR-10 simulation experiment.