Help the machine to help you: an evaluation in the wild of egocentric data cleaning via skeptical learning

TL;DR

This work evaluates Skeptical Learning (SKEL) in a real-world, longitudinal setting to clean egocentric contextual labels, focusing on location in university students using the iLog app. SKEL employs Gaussian Processes (GPs) to quantify uncertainty and guide user queries, with a two-phase, in-the-wild deployment that consolidates skeptical questions to reduce interruptions. Results show limited average gains over a non-skeptical baseline due to participant consistency and data gaps, though some individuals and longer studies may benefit from reduced annotation burden and improved data quality. The study demonstrates the feasibility of SKEL in mobile, in-the-wild data collection while highlighting design considerations such as scheduling, per-user hyperparameters, and the need for longer investigations and multi-modal inputs.

Abstract

Any digital personal assistant, whether used to support task performance, answer questions, or manage work and daily life, including fitness schedules, requires high-quality annotations to function properly. However, user annotations, whether actively produced or inferred from context (e.g., data from smartphone sensors), are often subject to errors and noise. Previous research on Skeptical Learning (SKEL) addressed the issue of noisy labels by comparing offline active annotations with passive data, allowing for an evaluation of annotation accuracy. However, this evaluation did not include confirmation from end-users, the best judges of their own context. In this study, we evaluate SKEL's performance in real-world conditions with actual users who can refine the input labels based on their current perspectives and needs. The study involves university students using the iLog mobile application on their devices over a period of four weeks. The results highlight the challenges of finding the right balance between user effort and data quality, as well as the potential benefits of using SKEL, which include reduced annotation effort and improved quality of collected data.

Figures (11)

• Figure 1: Research protocol. Blue boxes report the study phase and the white boxes the instruments.
• Figure 2: The three types of questions shown on the iLog app.
• Figure 3: Number of users who uploaded sensor data by day of the experiment. Red lines divide the three data collection phases.
• Figure 4: Number of time diary answers by hour of the day, divided by weekdays (left) and weekends (right). The answers are aggregated by main categories.
• Figure 5: The main category of the time diary answers over the first three weeks of the experiment. Rows: all users of the experiment. Columns: time interval of 30 minutes (i.e., annotation). White vertical lines denote each week.