BiFingerPose: Bimodal Finger Pose Estimation for Touch Devices

TL;DR

This paper tackles the limits of single-modality finger pose estimation on touch devices by proposing BiFingerPose, a bimodal framework that fuses capacitive images and under-screen fingerprint patches to recover full 2D and 3D finger pose. It introduces a trig-based angular representation with distribution vectors and maps 2D pose to 3D pose via a polynomial UV-to-3D transformation, enabling robust 360° yaw without hardware changes. Extensive experiments and a 12-person user study show BiFingerPose surpasses state-of-the-art single-modality methods by over 21% in pose accuracy, while delivering 2.5x higher task efficiency and 23% better user operation accuracy. The approach is designed for practical deployment on devices with under-screen sensors and is supported by prototypes and public code, highlighting its potential to broaden touch-based interaction and security applications.

Abstract

Finger pose offers promising opportunities to expand human computer interaction capability of touchscreen devices. Existing finger pose estimation algorithms that can be implemented in portable devices predominantly rely on capacitive images, which are currently limited to estimating pitch and yaw angles and exhibit reduced accuracy when processing large-angle inputs (especially when it is greater than 45 degrees). In this paper, we propose BiFingerPose, a novel bimodal based finger pose estimation algorithm capable of simultaneously and accurately predicting comprehensive finger pose information. A bimodal input is explored, including a capacitive image and a fingerprint patch obtained from the touchscreen with an under-screen fingerprint sensor. Our approach leads to reliable estimation of roll angle, which is not achievable using only a single modality. In addition, the prediction performance of other pose parameters has also been greatly improved. The evaluation of a 12-person user study on continuous and discrete interaction tasks further validated the advantages of our approach. Specifically, BiFingerPose outperforms previous SOTA methods with over 21% improvement in prediction performance, 2.5 times higher task completion efficiency, and 23% better user operation accuracy, demonstrating its practical superiority. Finally, we delineate the application space of finger pose with respect to enhancing authentication security and improving interactive experiences, and develop corresponding prototypes to showcase the interaction potential. Our code will be available at https://github.com/XiongjunGuan/DualFingerPose.

Figures (16)

• Figure 1: Examples of capacitive images and fingerprint patches in different touch poses.
• Figure 2: Users change their finger pose in any comfortable way to perform interactive operations on touch devices. Our BiFingerPose ultilizes capacitive image and fingerprint patch captured by touchscreen devices with under-screen fingerprint sensors to provide robust, precise and comprehensive finger pose estimation, which can be used for various applications. Notably, our solution eliminates the need for extra devices or the storage of pre-registered fingerprint information, ensuring both user privacy and a highly convenient experience.
• Figure 3: Definition and conversion process of three finger pose types. The precise mapping relationship between (a) 3D pose (roll, pitch, yaw of fingertip) and (b) 2D pose (position and angle of the fingerprint center in the screen coordinate system) can be established conveniently through (c) UV pose (position and angle of the contact center in the coordinate system of the normalized rolled fingerprint) introduced in this paper.
• Figure 4: Smartphone (a), IMU (b) and optical tracker (c) based data acquisition system. In particular, an optical fingerprint acquisition instrument is used to collect fingerprint images to obtain 2D poses in combination with fingerprint patches collected by a mobile phone. Samples of capacitive image & fingerprint patch (from smartphone) and plain / rolled fingerprint (from optical scanner) are shown in (d).
• Figure 5: Distributions of roll, pitch and yaw angles in PRF.