Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synesthesia of Vehicles: Tactile Data Synthesis from Visual Inputs

Rui Wang, Yaoguang Cao, Yuyi Chen, Jianyi Xu, Zhuoyang Li, Jiachen Shang, Shichun Yang

TL;DR

The paper addresses the gap in autonomous-vehicle sensing by enabling tactile perception from vision through the Synesthesia of Vehicles (SoV) framework. It combines a real-vehicle multi-modal dataset, a spatiotemporal alignment mechanism, and a latent diffusion-based visual-tactile synesthetic model (vtSyn) to synthesize tactile data from visual inputs without manual annotations. Across road types and lighting, vtSyn outperforms baselines in distributional fidelity, frequency-domain similarity, and downstream tactile-based classification, demonstrating potential for proactive vehicle control and safety. The work advances cross-modal sensing in AVs and lays groundwork for integrating tactile predictions into real-time driving decisions, with future work on adverse weather and system integration.

Abstract

Autonomous vehicles (AVs) rely on multi-modal fusion for safety, but current visual and optical sensors fail to detect road-induced excitations which are critical for vehicles' dynamic control. Inspired by human synesthesia, we propose the Synesthesia of Vehicles (SoV), a novel framework to predict tactile excitations from visual inputs for autonomous vehicles. We develop a cross-modal spatiotemporal alignment method to address temporal and spatial disparities. Furthermore, a visual-tactile synesthetic (VTSyn) generative model using latent diffusion is proposed for unsupervised high-quality tactile data synthesis. A real-vehicle perception system collected a multi-modal dataset across diverse road and lighting conditions. Extensive experiments show that VTSyn outperforms existing models in temporal, frequency, and classification performance, enhancing AV safety through proactive tactile perception.

Synesthesia of Vehicles: Tactile Data Synthesis from Visual Inputs

TL;DR

The paper addresses the gap in autonomous-vehicle sensing by enabling tactile perception from vision through the Synesthesia of Vehicles (SoV) framework. It combines a real-vehicle multi-modal dataset, a spatiotemporal alignment mechanism, and a latent diffusion-based visual-tactile synesthetic model (vtSyn) to synthesize tactile data from visual inputs without manual annotations. Across road types and lighting, vtSyn outperforms baselines in distributional fidelity, frequency-domain similarity, and downstream tactile-based classification, demonstrating potential for proactive vehicle control and safety. The work advances cross-modal sensing in AVs and lays groundwork for integrating tactile predictions into real-time driving decisions, with future work on adverse weather and system integration.

Abstract

Autonomous vehicles (AVs) rely on multi-modal fusion for safety, but current visual and optical sensors fail to detect road-induced excitations which are critical for vehicles' dynamic control. Inspired by human synesthesia, we propose the Synesthesia of Vehicles (SoV), a novel framework to predict tactile excitations from visual inputs for autonomous vehicles. We develop a cross-modal spatiotemporal alignment method to address temporal and spatial disparities. Furthermore, a visual-tactile synesthetic (VTSyn) generative model using latent diffusion is proposed for unsupervised high-quality tactile data synthesis. A real-vehicle perception system collected a multi-modal dataset across diverse road and lighting conditions. Extensive experiments show that VTSyn outperforms existing models in temporal, frequency, and classification performance, enhancing AV safety through proactive tactile perception.
Paper Structure (21 sections, 10 equations, 11 figures, 3 tables)

This paper contains 21 sections, 10 equations, 11 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORKS
  3. Synesthesia
  4. Synesthesia of Humans
  5. Synesthesia of Robots
  6. Cross-Modal Generation
  7. Methodology
  8. Architecture
  9. Data collection
  10. Spatiotemporal alignment method for visual and tactile data
  11. visual-tactile synesthetic generation model (vtSyn)
  12. VAE
  13. Feature extractor
  14. Diffusion process
  15. Experiments and Results
  16. ...and 6 more sections

Figures (11)

  • Figure 1: Synesthesia of Vehicles: inspired by human's Synesthesia, providing tactile perception of road ahead and helping vehicles to react proactively.
  • Figure 2: Visual-tactile synesthesia model for cross-modal generation.
  • Figure 3: The proposed visual-tactile perception system for autonomous vehicles.
  • Figure 4: The visual data of different roads on daytime and nighttime: (a)-(b): asphalt road at daytime and nighttime; (c)-(d): brick road at daytime and nighttime; (e)-(f): dirt road at daytime and nighttime; (g)-(h): cement road at daytime and nighttime; (i)-(j): muddy road at daytime and nighttime; (k)-(l): gravel road at daytime and nighttime.
  • Figure 5: The tactile data corresponding to different roads: (a) asphalt roads; (b) muddy roads; (c) gravel roads; (d) brick roads; (e) dirt roads; (f) cement roads.
  • ...and 6 more figures