Can Capacitive Touch Images Enhance Mobile Keyboard Decoding?

TL;DR

The results show that adding the heatmap into the input feature set led to 21.4% relative reduction of character error rates on average, compared to using the centroid alone, underline the promise of utilizing touch heatmaps for improving typing experience in mobile keyboards.

Abstract

Capacitive touch sensors capture the two-dimensional spatial profile (referred to as a touch heatmap) of a finger's contact with a mobile touchscreen. However, the research and design of touchscreen mobile keyboards -- one of the most speed and accuracy demanding touch interfaces -- has focused on the location of the touch centroid derived from the touch image heatmap as the input, discarding the rest of the raw spatial signals. In this paper, we investigate whether touch heatmaps can be leveraged to further improve the tap decoding accuracy for mobile touchscreen keyboards. Specifically, we developed and evaluated machine-learning models that interpret user taps by using the centroids and/or the heatmaps as their input and studied the contribution of the heatmaps to model performance. The results show that adding the heatmap into the input feature set led to 21.4% relative reduction of character error rates on average, compared to using the centroid alone. Furthermore, we conducted a live user study with the centroid-based and heatmap-based decoders built into Pixel 6 Pro devices and observed lower error rate, faster typing speed, and higher self-reported satisfaction score based on the heatmap-based decoder than the centroid-based decoder. These findings underline the promise of utilizing touch heatmaps for improving typing experience in mobile keyboards.

Figures (8)

• Figure 1: An example of the touch heatmap (the $16 \times 18$ colored grid laying on the keyboard layout) and the derived touch centroid (the red dot). In this example, the user aimed to type "c"; however, the derived touch centroid is on the key "x". Applying simple baselines (i.e., the On-key and the Distance baselines) would lead to an incorrect key prediction. Resolution scale: 1 pixel $\approx$ 0.05 mm.
• Figure 2: The distribution of touch centroids in the dataset from Study 1 pooled across all 24 participants. It is plotted on the QWERTY keyboard layout used in this study, which is the same layout as in the other two studies. The keyboard width ($W$), the keyboard height ($H$), the most common key width ($w$), and the most common key height ($h$) in pixels of the keyboard layout are also annotated in this figure. Resolution scale: 1 pixel $\approx$ 0.05 mm.
• Figure 3: An illustration of how to compute the heatmap overlap feature for a given key. The key boundary is represented by the red box, while the heatmap sensor array is represented by the grid the red box is on. $v_{ij}$ represents the intensity value of the heatmap cell at row $i$ and column $j$. The heatmap in this figure is made smaller ($3\times4$) than its actual size for clarity of illustration.
• Figure 4: The ($CH_o$) logistic regression model takes the centroid and the heatmap overlap vector as input and predicts probabilities of the candidate keys. Note that $W$ and $b$ are trained parameters of the model.
• Figure 5: Character distributions (from A to Z and SPACE, PERIOD on the x-axis) in the MacKenzie and Soukoreff corpus mackenzie2003phrase (top left), the final prompt pool for greedy selection (top right), the 90 selected prompts used for data collection (bottom left), and the processed main dataset used for training and evaluation (bottom right).