DipMe: Haptic Recognition of Granular Media for Tangible Interactive Applications

TL;DR

DipMe is introduced as a smart device to recognize the types of granular media in real time, which can be used to connect the granular materials in the physical world with various virtual content.

Abstract

While tangible user interface has shown its power in naturally interacting with rigid or soft objects, users cannot conveniently use different types of granular materials as the interaction media. We introduce DipMe as a smart device to recognize the types of granular media in real time, which can be used to connect the granular materials in the physical world with various virtual content. Other than vision-based solutions, we propose a dip operation of our device and exploit the haptic signals to recognize different types of granular materials. With modern machine learning tools, we find the haptic signals from different granular media are distinguishable by DipMe. With the online granular object recognition, we build several tangible interactive applications, demonstrating the effects of DipMe in perceiving granular materials and its potential in developing a tangible user interface with granular objects as the new media.

Figures (10)

• Figure 1: DipMe: system overview (left) and applications (right). A user is allowed to dip the device into different granular media. We collect the force and torque signals and use machine learning techniques to recognize the type of the granular material from the multichannel time series data. With the tracked motion of DipMe and the recognized type of granular material, we demonstrate several applications including a virtual drawing interface and a virtual music instrument built with DipMe.
• Figure 2: The hardware of measurement system. (a) The main modules of the system, in which a1-a4 make up DipMe, a5-a7 actuate the sensors and transmit the collected force signals to PC. a1-HTC ViVE Tracker 3.0, a2-handle side, a3-3D force sensor, a4-sensing side, a5-switching power supply, a6-data acquisition card, a7-signal transmitter. The left subfigure shows the exploded view of the sensor structure (a3): the top cover and conducting flange are made of lightweight and strong PEEK material. The four load cells are arranged in a square formation to form the combined sensing area. (b) The assistant modules of the system, b1 and b2-HTC VIVE Base station 1.0, b3-instrument calibration table, b4-granular media test box.
• Figure 3: Examples of LPF processing.
• Figure 4: Dipping test experiments and filtered force signals of 6 granular media. Nutrition Soil (NuSoil), whose particle size distribution is about 0$\sim$0.002mm, Millet with 0.007$\sim$0.033mm, Cement with 0.01$\sim$0.02mm bentz1999effects, Sand with 0.063$\sim$2mm kettler2001simplified, Mung with 3$\sim$4mm and Simulated Soil (SimuSoil) with 7$\sim$8mm.
• Figure 5: Illustration of the Multi-channel Encoder Model.