NMF-Based Analysis of Mobile Eye-Tracking Data

TL;DR

The paper addresses unsupervised identification of Areas of Interest (AOIs) in mobile eye-tracking data by applying nonnegative matrix factorization (NMF) to gaze-informed video frames. It introduces a preprocessing pipeline that crops image patches around gaze points, reduces temporal redundancy via fixations, and vectorizes frames into a matrix $X \in \mathbb{R}^{(3\cdot\tilde{n}_1\cdot\tilde{n}_2) \times (\tilde{m}_1+\dots+\tilde{m}_r)}$; NMF then computes $X = W H$ with $W \ge 0$, $H \ge 0$, yielding $k$ spatial components $w_j$ and temporal components $h_j$. Visualization ranks components by impact using $\|h_j\|_p$ and presents spatial images with temporal indicator plots for exploratory assessment. The method is demonstrated on nine mobile eye-tracking recordings of an art-gallery scene, extended to 27 recordings, showing successful identification of multiple AOIs and revealing scanpath patterns, while noting scalability constraints and potential future interactive enhancements.

Abstract

The depiction of scanpaths from mobile eye-tracking recordings by thumbnails from the stimulus allows the application of visual computing to detect areas of interest in an unsupervised way. We suggest using nonnegative matrix factorization (NMF) to identify such areas in stimuli. For a user-defined integer k, NMF produces an explainable decomposition into k components, each consisting of a spatial representation associated with a temporal indicator. In the context of multiple eye-tracking recordings, this leads to k spatial representations, where the temporal indicator highlights the appearance within recordings. The choice of k provides an opportunity to control the refinement of the decomposition, i.e., the number of areas to detect. We combine our NMF-based approach with visualization techniques to enable an exploratory analysis of multiple recordings. Finally, we demonstrate the usefulness of our approach with mobile eye-tracking data of an art gallery.

Figures (7)

• Figure 1: Our visual analysis technique applied to nine mobile eye-tracking recordings of a scene from an art gallery. Based on the underlying nonnegative matrix factorization (NMF), the preprocessed recordings are decomposed into eight spatiotemporal components, where each is described by a spatial representation, temporal indicator plots, and an impact. The combination of these two representations provides a spatiotemporal clustering of the recordings. To link the eight clusters to the recordings, a representative image from each recording is assigned to it based on the highest peak of the respective temporal indicator plots.
• Figure 2: Overview of our visually guided approach to identifying spatiotemporal features among multiple eye-tracking recordings: After preprocessing the recordings (using image patches, focusing on fixations, and vectorizing them into a matrix representation), the edited frames are represented in an overarching matrix that can be factorized via NMF to generate appropriate clusters. Finally, a clustered representation with interactive visualizations is established to provide an exploratory analysis of the eye-tracking recordings.
• Figure 3: Overview of our strategy to visualize components in our NMF-based visual analysis of eye-tracking recordings. The extracted spatiotemporal patterns belong to the most dominant component in \ref{['fig:teaser']}, where nine eye-tracking recordings are factorized into k = 8 components. The represented component consists of an impact value, a spatial representation, and associated temporal indicator plots. Furthermore, representative reference images from the recordings based on the highest peaks (enclosed by two gray lines in the temporal indicator plots) are shown. The reference images for each recording are displayed side by side and row by row, as the color of the image frames and temporal indicator plots demonstrate.
• Figure 4: Scene of Artwork Gallery with five paintings: Software Feathers (F), roboPix (R), Hough Arts (H), Bubble Hierarchies (B), and Frayed Cell Diagram (C). Each painting is accompanied by a plate with text description Text (T$_\text{p}$) with p$\mathbf{\in \{}$F, R, H, B, C$\mathbf{\}}$. The two walking directions LR and RL define the attendance order. The order between a painting and its text description induces a subpattern.
• Figure 5: Visual comparison of manually labeled AOIs in our NMF-based analysis of nine recordings of a scene from an art gallery. Manual labels are illustrated as small sample images of the five AOIs in the first four temporal indicator plots. This allows evaluating the NMF-based decomposition via their indicator plots. The remaining components, namely the spatial representation and the representative images, are depicted in \ref{['fig:teaser']}.