Disjoint Contrastive Regression Learning for Multi-Sourced Annotations

TL;DR

A novel contrastive regression framework is proposed to address the disjoint annotations problem, where each sample is labeled by only one annotator and multiple annotators work on disJoint subsets of the data.

Abstract

Large-scale datasets are important for the development of deep learning models. Such datasets usually require a heavy workload of annotations, which are extremely time-consuming and expensive. To accelerate the annotation procedure, multiple annotators may be employed to label different subsets of the data. However, the inconsistency and bias among different annotators are harmful to the model training, especially for qualitative and subjective tasks.To address this challenge, in this paper, we propose a novel contrastive regression framework to address the disjoint annotations problem, where each sample is labeled by only one annotator and multiple annotators work on disjoint subsets of the data. To take account of both the intra-annotator consistency and inter-annotator inconsistency, two strategies are employed.Firstly, a contrastive-based loss is applied to learn the relative ranking among different samples of the same annotator, with the assumption that the ranking of samples from the same annotator is unanimous. Secondly, we apply the gradient reversal layer to learn robust representations that are invariant to different annotators. Experiments on the facial expression prediction task, as well as the image quality assessment task, verify the effectiveness of our proposed framework.

Figures (4)

• Figure 1: Example of the intra-annotator consistency and inter-annotator inconsistency phenomenon. Each box represents a sample from the dataset. The blue color from dark to light represents the score from low to high. After labeling, the intra-annotator consistency remains, while the inter-annotator consistency is violated.
• Figure 2: The framework of disjoint contrastive regression (DCR). The input data is first fed to the embedding net $f_{\boldsymbol{\theta}}$. The score is then predicted by the regression head $g_{\boldsymbol{\phi}}$. The pairwise distance of the prediction is optimized by the supervision of disjoint ranking loss. In the meanwhile, the embeddings are fed to a second classification branch $h_{\boldsymbol{\psi}}$ that aims to classify different annotators. We adopt the gradient reversal layer (GRL) to learn the robust annotator-invariant embeddings.
• Figure 3: Selected examples with their corresponding scores, including the images and their corresponding scores, from the datasets. The images on the first line are samples for facial expression score prediction while those on the second line are for IQA.
• Figure 4: An example of the score distributions from four annotators of the TID2013 dataset.