QualitEye: Public and Privacy-preserving Gaze Data Quality Verification

Abstract

Gaze-based applications are increasingly advancing with the availability of large datasets but ensuring data quality presents a substantial challenge when collecting data at scale. It further requires different parties to collaborate, therefore, privacy concerns arise. We propose QualitEye--the first method for verifying image-based gaze data quality. QualitEye employs a new semantic representation of eye images that contains the information required for verification while excluding irrelevant information for better domain adaptation. QualitEye covers a public setting where parties can freely exchange data and a privacy-preserving setting where parties cannot reveal their raw data nor derive gaze features/labels of others with adapted private set intersection protocols. We evaluate QualitEye on the MPIIFaceGaze and GazeCapture datasets and achieve a high verification performance (with a small overhead in runtime for privacy-preserving versions). Hence, QualitEye paves the way for new gaze analysis methods at the intersection of machine learning, human-computer interaction, and cryptography.

Figures (10)

• Figure 1: To verify the image-based gaze data quality, source data owners $O$ (blue) and the reference party $R$ (e.g. a reliable source, red), first, disentangle the gaze direction and head pose features that correspond to the data labels, ignoring the cross-party irrelevant features (e.g. appearance) for a high domain adaptation performance instead of the raw pixel-wise data comparison. Then, features are semantically hashed for an efficient bit-wise comparison to obtain data-independent (i.e. not domain-specific), deterministic (i.e. produces the same outputs for similar semantics), and generative (i.e. learns the gaze data distribution) representations $z$ (shown in orange). Then, they compare the hash values and corresponding labels against each other. to find the (mis)matching data samples. C.f. Figure \ref{['fig:VAE']} for the shared semantic representation and Figures \ref{['fig:DH']} and \ref{['fig:OPRF']} for the comparison.
• Figure 2: To obtain the hashed semantic representations used for label-based data comparison, 1- green: eye images are encoded into a latent vector $z$. 2- red: Then, the gaze and head pose information are disentangled from the appearance via some transformations and a multi-layer perception (MLP). 3- blue: During training, the appearance, gaze, and pose are passed as a latent vector to a decoder that reconstructs the transformed eye images. 4- yellow: Finally, the gaze and head pose latent codes are hashed and are passed as inputs to the PSI protocol along with the corresponding labels.
• Figure 3: In QualitEye-V$_{1}$, both parties $O$ and $R$ exchange their elements as hashed values raised to their private keys. Then, they raise the other party's received values again to their private keys. $R$ then start the comparison to find the matching inputs. Note that, we further send the eye labels (e.g. gaze direction and head pose) as an additional encrypted payload to each element. In QualitEye-V$_{2}$, $O$ shuffles the second message to only reveal the cardinality of the intersection. In QualitEye-V$_{3}$, $R$ includes the first message, e.g. as a package, before the start of the protocol. In all versions, only the private keys are secret, and all other values are sent in the clear. Security is still guaranteed due to the hardness of the 'discrete logarithm problem', i.e. it is hard to infer the private keys diffie1976new.
• Figure 4: In QualitEye-V$_{4}$, each party computes two values per element (i.e. the PRF output of the hashed semantic representations). $R$ only selects one value per element (via cuckoo hashing). $O$ send his values (along with the encrypted blinded labels) to $R$. $R$ sends back all received information along with her encrypted blinded labels to $O$, who makes the comparison and finds the (mis)matching elements.
• Figure 5: A qualitative example of the correct predictions (green) and incorrect predictions (red) of the gaze direction verification. Each pair represents different aspects of appearance, e.g. different subjects, head poses, lighting conditions, glasses, makeup, genders, and race. Therefore, QualitEye can successfully disentangle the appearance code, removing its effect from the overall method.