Signal Processing for Haptic Surface Modeling: a Review

TL;DR

This work surveys the relatively underexplored area of haptic surface modeling and data representation within the haptic processing pipeline. It contrasts parametric and data-driven modeling, catalogs human-computer, machine, and mixed haptic signals, and inventories datasets and simulators that enable surface texture and material understanding. The authors identify a lack of standardization across data formats, representations, and evaluation protocols, and they propose open directions for standard pipelines, cross-modal integration, real-time processing, and perceptually grounded metrics. By clarifying the links between physical feature spaces and perceptual spaces, the paper highlights the potential impact on realistic haptic experiences in XR and robotic manipulation, and it calls for community-wide benchmarks and shared resources to accelerate progress.

Abstract

Haptic feedback has been integrated into Virtual and Augmented Reality, complementing acoustic and visual information and contributing to an all-round immersive experience in multiple fields, spanning from the medical domain to entertainment and gaming. Haptic technologies involve complex cross-disciplinary research that encompasses sensing, data representation, interactive rendering, perception, and quality of experience. The standard processing pipeline, consists of (I) sensing physical features in the real world using a transducer, (II) modeling and storing the collected information in some digital format, (III) communicating the information, and finally, (IV) rendering the haptic information through appropriate devices, thus producing a user experience (V) perceptually close to the original physical world. Among these areas, sensing, rendering and perception have been deeply investigated and are the subject of different comprehensive surveys available in the literature. Differently, research dealing with haptic surface modeling and data representation still lacks a comprehensive dissection. In this work, we aim at providing an overview on modeling and representation of haptic surfaces from a signal processing perspective, covering the aspects that lie in between haptic information acquisition on one side and rendering and perception on the other side. We analyze, categorize, and compare research papers that address the haptic surface modeling and data representation, pointing out existing gaps and possible research directions.

Figures (4)

• Figure 1: The haptic processing pipeline consists of 5 main steps: sensors to acquire the data, processing and modeling to store acquired data, communication to transmit information associated with the collected signals, haptic devices to render the haptic feedback to the user, and perception that studies the factor that influence the haptic experience of the user.
• Figure 2: The classic signal processing pipeline (top row) aims at mapping a physical phenomenon to a physical feature space such that a similar experience of the phenomenon can be delivered to the user in the perception feature space (bottom row). While the pipeline is well-established in both the visual and acoustic domains, it still lacks a common standard in the haptic domain. In the example in bottom row illustrating the section of a tree, the redline is the roughness, the green line is the bumpiness, the blue line represents the stiffness, and the dark gray line is the friction.
• Figure 3: Haptic surface modeling approaches can be categorized into parametric and data-driven methodologies. Furthermore, data-driven methods can be subdivided into human-computer haptics, which model haptic properties with a focus on human-based applications, and machine haptics, which model haptic properties specifically for robotic applications.
• Figure 4: Data-driven approaches require the collection of haptic signals, which can be either (a) mono-dimensional (taken from culbertson2012refined) or (b) vision-based (taken from yang2022touch). Mono-dimensional signals usually describe a single feature acquired by a sensor, like speed, force, and acceleration. Vision-based signals provide spatially-related 2D high-resolution information of a surface.