Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating Active Sampling for Hardness Classification with Vision-Based Tactile Sensors

Junyi Chen, Alap Kshirsagar, Frederik Heller, Mario Gómez Andreu, Boris Belousov, Tim Schneider, Lisa P. Y. Lin, Katja Doerschner, Knut Drewing, Jan Peters

TL;DR

This work tackles sample-efficient hardness classification using vision-based tactile sensors by framing hardness as a task amenable to information-theoretic active sampling. It compares three probabilistic classifiers and two uncertainty-based sampling strategies, evaluating them on robot-collected and human-collected datasets, with a human study for baseline context. Results show that uncertainty-driven active sampling, particularly the variance-based approach, yields higher accuracy and stability than random sampling, and that the best VBTS-based method can substantially exceed human performance on the same objects ($ ext{best} ightarrow 88.78\%$, ext{humans} ightarrow 48\%$). The study also demonstrates the role of reservoir sampling for scalable retraining and discusses limitations and avenues for multi-modal and multi-finger extensions to enhance robustness in real-world applications.

Abstract

One of the most important object properties that humans and robots perceive through touch is hardness. This paper investigates information-theoretic active sampling strategies for sample-efficient hardness classification with vision-based tactile sensors. We evaluate three probabilistic classifier models and two model-uncertainty-based sampling strategies on a robotic setup as well as on a previously published dataset of samples collected by human testers. Our findings indicate that the active sampling approaches, driven by uncertainty metrics, surpass a random sampling baseline in terms of accuracy and stability. Additionally, while in our human study, the participants achieve an average accuracy of 48.00%, our best approach achieves an average accuracy of 88.78% on the same set of objects, demonstrating the effectiveness of vision-based tactile sensors for object hardness classification.

Investigating Active Sampling for Hardness Classification with Vision-Based Tactile Sensors

TL;DR

This work tackles sample-efficient hardness classification using vision-based tactile sensors by framing hardness as a task amenable to information-theoretic active sampling. It compares three probabilistic classifiers and two uncertainty-based sampling strategies, evaluating them on robot-collected and human-collected datasets, with a human study for baseline context. Results show that uncertainty-driven active sampling, particularly the variance-based approach, yields higher accuracy and stability than random sampling, and that the best VBTS-based method can substantially exceed human performance on the same objects (, ext{humans} ightarrow 48\%$). The study also demonstrates the role of reservoir sampling for scalable retraining and discusses limitations and avenues for multi-modal and multi-finger extensions to enhance robustness in real-world applications.

Abstract

One of the most important object properties that humans and robots perceive through touch is hardness. This paper investigates information-theoretic active sampling strategies for sample-efficient hardness classification with vision-based tactile sensors. We evaluate three probabilistic classifier models and two model-uncertainty-based sampling strategies on a robotic setup as well as on a previously published dataset of samples collected by human testers. Our findings indicate that the active sampling approaches, driven by uncertainty metrics, surpass a random sampling baseline in terms of accuracy and stability. Additionally, while in our human study, the participants achieve an average accuracy of 48.00%, our best approach achieves an average accuracy of 88.78% on the same set of objects, demonstrating the effectiveness of vision-based tactile sensors for object hardness classification.
Paper Structure (19 sections, 3 equations, 9 figures, 1 table, 1 algorithm)

This paper contains 19 sections, 3 equations, 9 figures, 1 table, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. METHOD
  3. Classifier Input
  4. Model Architectures
  5. Active Sampling Strategies
  6. Retraining with Reservoir Sampling
  7. EVALUATION
  8. Datasets
  9. Our Dataset
  10. Yuan Dataset
  11. Hyperparameters
  12. Evaluation Metrics
  13. Model Baselines
  14. Effects of Dropout Rate
  15. Active Sampling Performance
  16. ...and 4 more sections

Figures (9)

  • Figure 1: The hardness classification task requires the agent to classify the test object into one of the reference classes based on hardness level. (a) The robot uses a GelSight Mini sensor mounted on its end-effector to explore the hardness of objects. The image captured by the sensor is shown in the inset. (b) Our dataset consists of GelSight Mini videos collected from five silicone objects of different shapes in increasing hardness from left to right. (c) In our human participant study, participants explored the test object (blue plate) and the reference objects (black plates) with their index fingers to compare hardness.
  • Figure 2: The model architectures of OpticalFlowNN, DINOv2NN and ConvLSTM.
  • Figure 3: Training and validation accuracies of the classifier models (see Sec. \ref{['sec:classifier_models']}) on our dataset (a and b) and the Yuan dataset (c and d) (see Sec. \ref{['sec:datasets']}). The accuracies are averaged over 50 runs and 90 runs, respectively, for the two datasets.
  • Figure 4: Mean test class accuracy and MAE, and their corresponding standard deviation over different dropout rates of DINOv2NN model on the Yuan Dataset. Savitzky-Golay filter is used to smooth the data points for visualization purposes.
  • Figure 5: Comparison of the classification accuracies and MAEs for the three strategies on our dataset. The shaded area indicates the standard deviation. In most iterations, the variance strategy achieves the highest accuracy and lowest MAE, closely followed by the entropy strategy.
  • ...and 4 more figures