HaloTouch: Using IR Multi-path Interference to Support Touch Interactions With General Surfaces

TL;DR

HaloTouch tackles the problem of enabling high-precision touch input on everyday surfaces without instrumenting those surfaces. It leverages multipath interference in commodity time-of-flight depth cameras to produce a Halo signal around the fingertip, enabling touch, hover, and pressure sensing with millimeter-scale spatial accuracy and a latency of $150\text{ ms}$. The approach combines a hardware setup (depth camera and projector) with a signal-processing and learning-based calibration pipeline, including a $20\text{ s}$ per-user calibration, to achieve $99.2\%$ touch-down accuracy across materials. This work demonstrates robust interaction on uninstrumented surfaces and enables practical applications such as a virtual keyboard, gesture-driven drumming, and pressure-sensitive painting, suggesting significant potential for ad hoc, surface-agnostic interaction in real-world settings.

Abstract

Sensing touch on arbitrary surfaces has long been a goal of ubiquitous computing, but often requires instrumenting the surface. Depth camera-based systems have emerged as a promising solution for minimizing instrumentation, but at the cost of high touch-down detection error rates, high touch latency, and high minimum hover distance, limiting them to basic tasks. We developed HaloTouch, a vision-based system which exploits a multipath interference effect from an off-the-shelf time-of-flight depth camera to enable fast, accurate touch interactions on general surfaces. HaloTouch achieves a 99.2% touch-down detection accuracy across various materials, with a motion-to-photon latency of 150 ms. With a brief (20s) user-specific calibration, HaloTouch supports millimeter-accurate hover sensing as well as continuous pressure sensing. We conducted a user study with 12 participants, including a typing task demonstrating text input at 26.3 AWPM. HaloTouch shows promise for more robust, dynamic touch interactions without instrumenting surfaces or adding hardware to users.

Figures (19)

• Figure 1: Radar charts comparing eight touch input systems (ElectrickZhang2017Electrick, ShadowTouchLiang2023ShadowTouch, Shi et al.Shi2020FingerIMU, TapLightStreli2023LightSpeckle, DirectXiao2016Direct, FarOutShen2021Farout, MicroPressDobinson2022MicroPress, HaloTouch) across metrics like Accuracy, Latency, Material Compatibility
• Figure 2: Multi-path Interference example scene: (a) Top view of Multi-path interference when camera faces a corner. (b) Side view of multi-path interference when camera faces a hand. (c) Top view of multi-path interference when camera faces a hand.
• Figure 3: Software: a touch detection pipeline using a Kinect Azure depth camera. It integrates IR and RGB streams with hand tracking and signal processing to isolate fingertip signals, correct noise, and detect touch events
• Figure 4: HaloTouch State Machine for Signal Correction, Calibration, and Touch Detection. This diagram outlines the signal processing pipeline for HaloTouch, structured into three key sections: Signal Correction (Sec 4.3.4), Calibration (Sec 4.3.5), and Touch Detection (Sec 4.3.6). The state machine transitions between Idle, Hover, Subtle, Touch, and Pressure states based on corrected halo signal values. The Touch Detection module predicts proximity, touch-down, and pressure values
• Figure 5: Four calibration states. First row: typical finger to surface distances for each state. Second row: halo effect visualization for each state.