Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarios

Ning Cheng, Jinan Xu, Jialing Chen, Wenjuan Han

TL;DR

This work tackles the challenge of enabling commonsense reasoning in open-ended physical environments by addressing modality gaps between touch and language and the scarcity of open-ended tactile data. It introduces SToLa, a self-adaptive touch-language framework that embeds Mixture of Experts inside a language model to dynamically handle tactile and textual tokens, guided by a two-stage training strategy. A new open-ended tactile benchmark, TactileBench, extends beyond prior datasets with eight physical properties, four interactive characteristics, and free-form prompts, enabling more realistic tactile reasoning. Empirical results show that SToLa achieves competitive or superior performance versus larger baselines on PhysiCLeAR and TactileBench, validating the MoE-based multimodal management approach and the practical utility of open-ended tactile commonsense reasoning.

Abstract

This paper explores the challenges of integrating tactile sensing into intelligent systems for multimodal reasoning, particularly in enabling commonsense reasoning about the open-ended physical world. We identify two key challenges: modality discrepancy, where existing large touch-language models often treat touch as a mere sub-modality of language, and open-ended tactile data scarcity, where current datasets lack the diversity, open-endness and complexity needed for reasoning. To overcome these challenges, we introduce SToLa, a Self-Adaptive Touch-Language framework. SToLa utilizes Mixture of Experts (MoE) to dynamically process, unify, and manage tactile and language modalities, capturing their unique characteristics. Crucially, we also present a comprehensive tactile commonsense reasoning dataset and benchmark featuring free-form questions and responses, 8 physical properties, 4 interactive characteristics, and diverse commonsense knowledge. Experiments show SToLa exhibits competitive performance compared to existing models on the PhysiCLeAR benchmark and self-constructed datasets, proving the effectiveness of the Mixture of Experts architecture in multimodal management and the performance advantages for open-scenario tactile commonsense reasoning tasks.

SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarios

TL;DR

This work tackles the challenge of enabling commonsense reasoning in open-ended physical environments by addressing modality gaps between touch and language and the scarcity of open-ended tactile data. It introduces SToLa, a self-adaptive touch-language framework that embeds Mixture of Experts inside a language model to dynamically handle tactile and textual tokens, guided by a two-stage training strategy. A new open-ended tactile benchmark, TactileBench, extends beyond prior datasets with eight physical properties, four interactive characteristics, and free-form prompts, enabling more realistic tactile reasoning. Empirical results show that SToLa achieves competitive or superior performance versus larger baselines on PhysiCLeAR and TactileBench, validating the MoE-based multimodal management approach and the practical utility of open-ended tactile commonsense reasoning.

Abstract

This paper explores the challenges of integrating tactile sensing into intelligent systems for multimodal reasoning, particularly in enabling commonsense reasoning about the open-ended physical world. We identify two key challenges: modality discrepancy, where existing large touch-language models often treat touch as a mere sub-modality of language, and open-ended tactile data scarcity, where current datasets lack the diversity, open-endness and complexity needed for reasoning. To overcome these challenges, we introduce SToLa, a Self-Adaptive Touch-Language framework. SToLa utilizes Mixture of Experts (MoE) to dynamically process, unify, and manage tactile and language modalities, capturing their unique characteristics. Crucially, we also present a comprehensive tactile commonsense reasoning dataset and benchmark featuring free-form questions and responses, 8 physical properties, 4 interactive characteristics, and diverse commonsense knowledge. Experiments show SToLa exhibits competitive performance compared to existing models on the PhysiCLeAR benchmark and self-constructed datasets, proving the effectiveness of the Mixture of Experts architecture in multimodal management and the performance advantages for open-scenario tactile commonsense reasoning tasks.
Paper Structure (18 sections, 9 equations, 5 figures, 4 tables)

This paper contains 18 sections, 9 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tactile Commonsense Reasoning
  4. Mixture of Experts
  5. Method
  6. Model Architecture
  7. Two-Stage Training Strategy
  8. TactileBench
  9. Task Category
  10. Data Construction
  11. Data Statistics
  12. Evaluation Metrics
  13. Experiments
  14. Implementation Details
  15. Results
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Comparison of our framework with prior arts in terms of tactile commonsense reasoning datasets and model capability. (a) We compare our proposed TactileBench dataset and the widely recognized tactile reasoning dataset, PhysiCLeARyu2024octopi. Unlike PhysiCLeAR, TactileBench offers broader coverage dimensions, expanding physical attributes from 3 to 8. We also add 4 new interaction characteristics. Additionally, instead of relying on templated designs for specific tasks, we focus on open-ended tasks with content format in a free-form manner. (b) We visualize the outstanding performance of the proposed model SToLa, highlighting four subtasks (PC, PSR, POM, PSS, and OPD) of PhysiCLeAR, and three categories (FPU, TIP, and CDR) belonging to TactileBench. PC: Property Comparison. PSS: Property Superlative Selection. POM: Property-object Matching. PSR: Property Scenario Reasoning. OPD: Object Property Description. FPU: Fundamental Property Understanding. TIP: Tactile Interaction Perception. CDR: Commonsense-Driven Reasoning. (c) Qualitative comparison of responses from different models. Errors are highlighted in red, while correct parts are emphasized in blue. For visualization, we include the visual signals of the tactile input (labeled as "Reference"), even though it's not used for the model.
  • Figure 2: SToLa framework. Our framework consists of a touch encoder, a touch-language adapter, and a LLM. The training process follows a two-stage strategy. Stage I: We train only the touch-language adapter, allowing the LLM to adapt to tactile inputs—either static tactile images with spatial details or dynamic tactile sequences with spatiotemporal information. Stage II: The weights from Stage I are copied, keeping the touch encoder unchanged. The self-attention module of the LLM is fine-tuned using LoRA, while the FFN is upcycled from dense to sparse. Notably, we do not adjust the word embedding layer throughout the process.
  • Figure 3: Distribution of expert loads across tactile and textual modality inputs.
  • Figure 4: Distribution of modalities across different experts.
  • Figure 5: Visualization of activated pathways.