Container Localisation and Mass Estimation with an RGB-D Camera

TL;DR

This work tackles the problem of estimating the empty mass of a container manipulated by a person in RGB-D video without relying on the container's content. It introduces a fixed-frontal RGB-D approach that localises the manipulated container by detecting multiple candidates with Mask R-CNN, selects up to K nearest depth patches, and uses a lightweight CNN to predict the empty mass per patch, aggregating predictions by averaging. The model combines RGB information with simple geometric cues (aspect ratios and distance) and achieves a 533k-parameter regression network that is trained on the CCM dataset with extensive data augmentation. Results show competitive performance under varying lighting and filling conditions (71.08% score on CCM), while highlighting generalization limitations to unseen container types, especially cups, and suggesting avenues for multi-task learning and audio-visual fusion to improve robustness and applicability in real-time HRI settings.

Abstract

In the research area of human-robot interactions, the automatic estimation of the mass of a container manipulated by a person leveraging only visual information is a challenging task. The main challenges consist of occlusions, different filling materials and lighting conditions. The mass of an object constitutes key information for the robot to correctly regulate the force required to grasp the container. We propose a single RGB-D camera-based method to locate a manipulated container and estimate its empty mass i.e., independently of the presence of the content. The method first automatically selects a number of candidate containers based on the distance with the fixed frontal view, then averages the mass predictions of a lightweight model to provide the final estimation. Results on the CORSMAL Containers Manipulation dataset show that the proposed method estimates empty container mass obtaining a score of 71.08% under different lighting or filling conditions.

Figures (4)

• Figure 1: Block diagram of the proposed approach. For each frame, containers are detected, then $K$ nearest patches are selected leveraging the raw depth maps considered in the segmentation mask coordinates. The empty mass of each patch ($m$) is predicted by the model which takes as input the RGB patch and triplets of values: aspect ratio width ($a$) and height ($b$), and average distance ($d$). The final empty mass estimation ($\hat{m}$) is the average of $K$ mass predictions.
• Figure 2: Sample patches of the extracted dataset. Black padding is applied before resizing to keep the same aspect ratio.
• Figure 3: 3-fold cross-validation setups ($F1$, $F2$, $F3$) of the CCM training set. Each fold selects videos from one instance of each container type as test set (), while videos belonging to the other instances are used as training set ().
• Figure 4: Analysis per container type of 3-fold cross validation and random cross-validation of our proposed model for container empty mass estimation. Top: testing score $s$ in percentage. Bottom: mean of relative absolute error $\epsilon$. The maximum y-axis value is set to 10 for visualization purpose, the actual value for cup in F1 is 22.952. Legend: cup, glass, boxtotal