The Effects of Selected Object Features on a Pick-and-Place Task: a Human Multimodal Dataset

TL;DR

This work presents a multimodal dataset to study how object properties, specifically weight and liquid content, modulate human pick-and-place kinematics. The controlled experiment with 15 participants across 80 trials per person (1200 actions) collects synchronized motion capture, inertial, and camera data as subjects manipulate identical cups under four conditions. A proof-of-concept using LSTM classifiers demonstrates that carefulness can be reliably inferred from each sensor modality (≈91% accuracy), while weight is harder to classify, highlighting the dominant influence of liquid content on motion strategies. The dataset provides a valuable resource for investigation of intention recognition and motion generation in human-robot interaction, enabling cross-modal comparisons and future extensions to richer carefulness notions and applications in robotics.

Abstract

We propose a dataset to study the influence of object-specific characteristics on human pick-and-place movements and compare the quality of the motion kinematics extracted by various sensors. This dataset is also suitable for promoting a broader discussion on general learning problems in the hand-object interaction domain, such as intention recognition or motion generation with applications in the Robotics field. The dataset consists of the recordings of 15 subjects performing 80 repetitions of a pick-and-place action under various experimental conditions, for a total of 1200 pick-and-places. The data has been collected thanks to a multimodal setup composed of multiple cameras, observing the actions from different perspectives, a motion capture system, and a wrist-worn inertial measurement unit. All the objects manipulated in the experiments are identical in shape, size, and appearance but differ in weight and liquid filling, which influences the carefulness required for their handling.

Figures (8)

• Figure 1: The three points of view of the experiment: the resting position as seen by the iCub robot (\ref{['fig:restpos_robot']}), a transportation movement towards the right shelf (\ref{['fig:shelfpos_lat']}), and the positioning of a glass on the scale (\ref{['fig:scalepos_back']}). In (\ref{['fig:frontal']}) the labels identify the 8 positions on the shelves.
• Figure 2: Top view of the experimental setup. In dark grey the cameras from the Optotrak motion capture system used to detect the active markers, while in black the two high resolution cameras.
• Figure 3: The structure of each trial falls within two possibilities, whereby the main action is the glass transportation: in \ref{['fig:fromshelf']} are shown the steps for taking a glass from the shelves and placing it on the scale, while in \ref{['fig:fromscale']} those for putting back the glass from the scale to the shelf.
• Figure 4: Marker positions on the participant's right arm and hand. The smartwatch equipped with inertial sensors is visible on the wrist.
• Figure 5: Box plots of each sensor recording frequency. The red lines represent the medians, the blue rectangles limit the 25th and 75th percentiles