Learning under Temporal Label Noise

TL;DR

This work proposes and formalizes temporal label noise, an unstudied problem for sequential classification of time series, and proposes methods to train noise-tolerant classifiers by estimating the temporal label noise function directly from data.

Abstract

Many time series classification tasks, where labels vary over time, are affected by label noise that also varies over time. Such noise can cause label quality to improve, worsen, or periodically change over time. We first propose and formalize temporal label noise, an unstudied problem for sequential classification of time series. In this setting, multiple labels are recorded over time while being corrupted by a time-dependent noise function. We first demonstrate the importance of modeling the temporal nature of the label noise function and how existing methods will consistently underperform. We then propose methods to train noise-tolerant classifiers by estimating the temporal label noise function directly from data. We show that our methods lead to state-of-the-art performance under diverse types of temporal label noise on real-world datasets

Figures (11)

• Figure 1: In time series tasks, label quality can vary over time due to temporal label noise. Existing methods assume noise can perform poorly as they assume a static noise model over time. We propose that accurate modeling of temporal label noise can improve performance. We demonstrate this by showing performance improvements on an activity recognition task where we compare reconstruction error and accuracy between static and temporal methods subject to 30% temporal label noise across 10 draws (see results for moving dataset in \ref{['Appendix::Results']} for details).
• Figure 2: Comparison of the ground truth unseen temporal noise function ${{\bm{Q}}}(t)$ and its estimate $\hat{{\bm{Q}}}(t)$ from each Temporal method on the moving data. We show the noise rate for the negative class only for clarity. We show results for Periodic and Mixed noise regimes. As shown, our Temporal methods can learn the true label noise function across different noise patterns. The resulting noise models have lower reconstruction error and are superior to static approaches.
• Figure 3: Temporal effects in label noise in a real-world stress detection task. We show noise rates when individuals are stressed (left) and not stressed (right). True noise rate is the average disagreement rates between clean and noisy labels over time. We can see clear, temporal label noise patterns -- our temporal label noise methods do a superior job of approximating it.
• Figure 4: Temporal label noise function ${\bm{Q}}(t)$ used in the experiments. We present six examples for binary classification task (from top-left clockwise): static, decay periodic, mixed, linear, growth. Each plot shows the off-diagonal entries of various parameterized forms of ${\bm{Q}}(t)$.
• Figure 5: Comparing performance of models trained with forward temporal loss vs no noise correction on synth with varying degrees of temporal label noise using either the true temporal noise function (Temporal) or the average temporal noise function (Static). Error bars are st. dev. over 10 runs.

Theorems & Definitions (4)

• Definition 1
• Definition 2
• Proposition 1
• proof