Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HaptoFloater: Visuo-Haptic Augmented Reality by Embedding Imperceptible Color Vibration Signals for Tactile Display Control in a Mid-Air Image

Rina Nagano, Takahiro Kinoshita, Shingo Hattori, Yuichi Hiroi, Yuta Itoh, Takefumi Hiraki

TL;DR

HaptoFloater proposes a visuo-haptic AR approach that embeds imperceptible color vibrations into mid-air images to drive a finger-worn haptic device, enabling low-latency tactile feedback without relying on external tracking. The system couples a MMAP-based mid-air display with color-vibration image generation and a RGB photodiode–driven tactile actuator, achieving a worst-case latency around 59.5 ms and a visual-haptic tolerance near 110 ms. User studies confirm that texture-aware tactile feedback enhances realism and that latency tolerance exceeds the device’s latency, supporting practical use in mid-air touch panels, texture design, and museum exhibits. The work offers a scalable path for rich VHAR interactions by leveraging imperceptible color modulation and wearable sensing to synchronize light-based visuals with tactile feedback.

Abstract

We propose HaptoFloater, a low-latency mid-air visuo-haptic augmented reality (VHAR) system that utilizes imperceptible color vibrations. When adding tactile stimuli to the visual information of a mid-air image, the user should not perceive the latency between the tactile and visual information. However, conventional tactile presentation methods for mid-air images, based on camera-detected fingertip positioning, introduce latency due to image processing and communication. To mitigate this latency, we use a color vibration technique; humans cannot perceive the vibration when the display alternates between two different color stimuli at a frequency of 25 Hz or higher. In our system, we embed this imperceptible color vibration into the mid-air image formed by a micromirror array plate, and a photodiode on the fingertip device directly detects this color vibration to provide tactile stimulation. Thus, our system allows for the tactile perception of multiple patterns on a mid-air image in 59.5 ms. In addition, we evaluate the visual-haptic delay tolerance on a mid-air display using our VHAR system and a tactile actuator with a single pattern and faster response time. The results of our user study indicate a visual-haptic delay tolerance of 110.6 ms, which is considerably larger than the latency associated with systems using multiple tactile patterns.

HaptoFloater: Visuo-Haptic Augmented Reality by Embedding Imperceptible Color Vibration Signals for Tactile Display Control in a Mid-Air Image

TL;DR

HaptoFloater proposes a visuo-haptic AR approach that embeds imperceptible color vibrations into mid-air images to drive a finger-worn haptic device, enabling low-latency tactile feedback without relying on external tracking. The system couples a MMAP-based mid-air display with color-vibration image generation and a RGB photodiode–driven tactile actuator, achieving a worst-case latency around 59.5 ms and a visual-haptic tolerance near 110 ms. User studies confirm that texture-aware tactile feedback enhances realism and that latency tolerance exceeds the device’s latency, supporting practical use in mid-air touch panels, texture design, and museum exhibits. The work offers a scalable path for rich VHAR interactions by leveraging imperceptible color modulation and wearable sensing to synchronize light-based visuals with tactile feedback.

Abstract

We propose HaptoFloater, a low-latency mid-air visuo-haptic augmented reality (VHAR) system that utilizes imperceptible color vibrations. When adding tactile stimuli to the visual information of a mid-air image, the user should not perceive the latency between the tactile and visual information. However, conventional tactile presentation methods for mid-air images, based on camera-detected fingertip positioning, introduce latency due to image processing and communication. To mitigate this latency, we use a color vibration technique; humans cannot perceive the vibration when the display alternates between two different color stimuli at a frequency of 25 Hz or higher. In our system, we embed this imperceptible color vibration into the mid-air image formed by a micromirror array plate, and a photodiode on the fingertip device directly detects this color vibration to provide tactile stimulation. Thus, our system allows for the tactile perception of multiple patterns on a mid-air image in 59.5 ms. In addition, we evaluate the visual-haptic delay tolerance on a mid-air display using our VHAR system and a tactile actuator with a single pattern and faster response time. The results of our user study indicate a visual-haptic delay tolerance of 110.6 ms, which is considerably larger than the latency associated with systems using multiple tactile patterns.
Paper Structure (31 sections, 2 equations, 10 figures, 1 table)

This paper contains 31 sections, 2 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Haptic devices in VHAR
  4. Color vibration for embedding information
  5. Tactile feedback on mid-air image displays
  6. Human perception characteristics for visuo-haptic latency tolerance
  7. HaptoFloater
  8. Mid-air image display
  9. Generating color vibration image
  10. Finger-worn haptic device
  11. Light-receiving sensor
  12. Tactile actuator
  13. Latency estimation
  14. Evaluation
  15. Worst-case latency of haptic device
  16. ...and 16 more sections

Figures (10)

  • Figure 1: System principle of HaptoFloater. (Left) Processing flow of visual and tactile information. (Right) Schematic illustration of the tactile feedback mechanism.
  • Figure 2: Mid-air display hardware of HaptoFloater. (a) Images on the LCD display screen appear in the air, symmetrical to the MMAP and the vision control film, called Louver Film (LF). (b) Touching the mid-air image with a finger-worn haptic device.
  • Figure 3: (a) Hardware configuration of HaptoFloater consisting of a finger-worn haptic device and a controller unit worn on the arm. (b) The controller unit consists of an audio module that transmits tactile vibrations and a microcontroller. (c) The finger-worn haptic device consists of an actuator and light-receiving sensor board using RGB photodiodes.
  • Figure 4: Configuration diagram of light-receiving sensor.
  • Figure 5: Experimental situation during the user study on latency perception. Although the figure shows the room with the lighting on for clarity, the actual study was conducted with dimmed lighting.
  • ...and 5 more figures