OmniVaT: Single Domain Generalization for Multimodal Visual-Tactile Learning

TL;DR

OmniVaT addresses single-domain generalization for multimodal visual-tactile learning by projecting VIS, TAC, and LANG features into a unified embedding-frequency space using a multimodal fractional Fourier adapter (MFFA). A discrete tree generation (DTG) module then diversifies fractional representations to robustly handle unseen domain shifts. The approach is trained with multimodal alignment, node diversity, and cross-entropy losses, and achieves state-of-the-art generalization across eight VIS-TAC domains, substantially reducing modality and domain gaps. This framework demonstrates strong cross-domain performance with real-time potential and provides a blueprint for extending single-domain generalization to other multimodal settings.

Abstract

Visual-tactile learning (VTL) enables embodied agents to perceive the physical world by integrating visual (VIS) and tactile (TAC) sensors. However, VTL still suffers from modality discrepancies between VIS and TAC images, as well as domain gaps caused by non-standardized tactile sensors and inconsistent data collection procedures. We formulate these challenges as a new task, termed single domain generalization for multimodal VTL (SDG-VTL). In this paper, we propose an OmniVaT framework that, for the first time, successfully addresses this task. On the one hand, OmniVaT integrates a multimodal fractional Fourier adapter (MFFA) to map VIS and TAC embeddings into a unified embedding-frequency space, thereby effectively mitigating the modality gap without multi-domain training data or careful cross-modal fusion strategies. On the other hand, it also incorporates a discrete tree generation (DTG) module that obtains diverse and reliable multimodal fractional representations through a hierarchical tree structure, thereby enhancing its adaptivity to fluctuating domain shifts in unseen domains. Extensive experiments demonstrate the superior cross-domain generalization performance of OmniVaT on the SDG-VTL task.

Figures (5)

• Figure 1: (a) Visualization of various VIS-TAC domains (e.g., , metal objects),  (b) Single-to-single domain methods gao2022objectfolderyang2022touch, (c) Multiple-to-single domain method feng2025anytouch, (d) Our single-to-multiple domain method.
• Figure 2: Embedding ($p=0$) or frequency ($p=1$) processing causes overlapping projections and inter-class confusion. However, the fractional Fourier transform (FrFT, $p=a$) provides a unified embedding-frequency representation that separates features from different classes more effectively. Circles and triangles represent VIS and TAC features, respectively.
• Figure 3: Overview of the proposed OmniVaT, including frozen multimodal encoders, a multimodal fractional Fourier adapter (MFFA) module with the fractional Fourier attention (FrATT) mechanism, and a discrete tree generation (DTG) module. The OmniVaT is jointly optimized by a multimodal alignment (MMA) loss, a node diversity (NOD) loss, and a cross-entropy (CE) loss. The frozen encoders are derived from CLIP radford2021learning. Both the MFFA and DTG modules share the same set of parameters across modalities. Since the LANG embedding contains class information, we use it only during training.
• Figure 4: The influence of the fractional order $p$ and the depth $R$ of tree on the TAG $\rightarrow$ X task. When $p$ = 0.0 or 1.0, it is fixed; otherwise, it is learnable.
• Figure 5: The influence of the extension parameter $E$ and the balance weight $\lambda$ on the TAG $\rightarrow$ X task.