TactStyle: Generating Tactile Textures with Generative AI for Digital Fabrication

TL;DR

TactStyle addresses the gap in AI-driven 3D textured stylization by learning a heightfield translator that maps texture images to tactile microgeometry and applying it to 3D models via UV mapping. By decoupling color and geometry stylization and jointly optimizing them, the system produces models whose tactile feel closely matches original textures and aligns with visual expectations. A formative study shows traditional geometry stylization fails to replicate heightfields, while a perception study with 15 participants demonstrates that TactStyle better replicates original tactile descriptors and resembles visual cues for several properties. The work demonstrates practical applications in decor, accessories, education, and assistive devices and outlines future work on richer datasets and material-aware tactile modeling.

Abstract

Recent work in Generative AI enables the stylization of 3D models based on image prompts. However, these methods do not incorporate tactile information, leading to designs that lack the expected tactile properties. We present TactStyle, a system that allows creators to stylize 3D models with images while incorporating the expected tactile properties. TactStyle accomplishes this using a modified image-generation model fine-tuned to generate heightfields for given surface textures. By optimizing 3D model surfaces to embody a generated texture, TactStyle creates models that match the desired style and replicate the tactile experience. We utilize a large-scale dataset of textures to train our texture generation model. In a psychophysical experiment, we evaluate the tactile qualities of a set of 3D-printed original textures and TactStyle's generated textures. Our results show that TactStyle successfully generates a wide range of tactile features from a single image input, enabling a novel approach to haptic design.

Figures (8)

• Figure 1: The boxplot shows the distribution of Root Mean Square (RMS) values for original and stylized textures, representing surface roughness. The original textures exhibit a wider range of RMS values, indicating higher variability in surface roughness. In contrast, the stylized textures have consistently higher RMS values with less variability, indicating rougher and more uniform surfaces as a result of the stylization process.
• Figure 2: TactStyle augments traditional 3D model stylization techniques by introducing a novel geometry stylization module that replicates the tactile properties of textures based on user input. (a) The system takes an input model and a stylization prompt (e.g., an image of a texture) and applies two separate stylization processes: (1) Color Stylization and (2) Geometry Stylization. The color stylization modifies the model's visual appearance, while the geometry stylization alters its surface to reflect tactile properties. The two modules operate in tandem, creating a stylized 3D model that replicates both the visual and tactile aspects of the texture. (b) The geometry stylization module uses a variational autoencoder (VAE) to generate heightfields from texture images, which are then applied to modify the model’s surface geometry, enabling co-optimization of geometry and color for a unified tactile and visual experience.
• Figure 3: TactStyle's user interface, implemented as a Blender plugin, allows users to load (a) original 3D model and (b) stylize with image prompts. In order to use TactStyle, the user (c) loads the model, (d) uploads an image of their desired texture, (e) optionally adjust the Texture Magnification Factor to control the level of height displacement applied on the 3D model. (f) Finally, the user clicks the "Stylize" button, which starts the stylization process using TactStyle’s integrated color and geometry stylization modules.
• Figure 4: Quantitative Comparison of TactStyle and Stylized Textures, and original heightfields. (a) Comparison of RMS values for Original, Stylized, and TactStyle textures, demonstrating that TactStyle replicates surface roughness more closely to the Original textures. (b) Box plot of MSE loss between the Original textures and the Stylized and TactStyle textures. TactStyle exhibits significantly lower MSE compared to the stylized method, indicating more accurate texture replication. (****p < 0.0001)
• Figure 5: 3D-Printed samples of 15 textures from our test set used in perception study: We created four different sets for our perception study: the 'visual set', 'original set', 'TactStyle set', and the 'stylization set'. The original set was created with the heightfield associated with the texture and served as the groundtruth. The reconstructed set was created using TactStyle, with the texture image as input. The visual set was created using printed texture images. Finally, the stylization (baseline) set was created using Style2Fab, using the texture image as input.