UNIC: Learning Unified Multimodal Extrinsic Contact Estimation

TL;DR

UNIC tackles the problem of extrinsic contact estimation without relying on priors or camera calibration by learning a unified, multimodal representation that encodes visual data in the camera frame and fuses it with proprioceptive and tactile signals. It introduces a prior-free scene-affordance map to represent diverse contact types and chains, and a masked multimodal fusion strategy to enhance robustness to missing modalities and varying sensor availability. Empirical results show UNIC achieves a 9.6 mm average Chamfer distance on unseen contact locations, generalizes to unseen objects, and maintains real-time performance (>600 Hz) even under test-time modality dropouts, demonstrating practicality for real-world, contact-rich manipulation. The work highlights strong generalization, robustness, and potential for integration with manipulation policies, suggesting scalable paths through simulation, foundation-models, and policy learning to broaden deployment.

Abstract

Contact-rich manipulation requires reliable estimation of extrinsic contacts-the interactions between a grasped object and its environment which provide essential contextual information for planning, control, and policy learning. However, existing approaches often rely on restrictive assumptions, such as predefined contact types, fixed grasp configurations, or camera calibration, that hinder generalization to novel objects and deployment in unstructured environments. In this paper, we present UNIC, a unified multimodal framework for extrinsic contact estimation that operates without any prior knowledge or camera calibration. UNIC directly encodes visual observations in the camera frame and integrates them with proprioceptive and tactile modalities in a fully data-driven manner. It introduces a unified contact representation based on scene affordance maps that captures diverse contact formations and employs a multimodal fusion mechanism with random masking, enabling robust multimodal representation learning. Extensive experiments demonstrate that UNIC performs reliably. It achieves a 9.6 mm average Chamfer distance error on unseen contact locations, performs well on unseen objects, remains robust under missing modalities, and adapts to dynamic camera viewpoints. These results establish extrinsic contact estimation as a practical and versatile capability for contact-rich manipulation.

Figures (8)

• Figure 1: UNIC leverages multimodal inputs to estimate a unified contact affordance map that captures diverse forms of extrinsic contact, including complex interaction chains such as gripper–object–object–environment. Throughout this process, UNIC does not rely on prior knowledge or camera calibration. At deployment, it remains effective even under missing modalities, adapts to dynamic camera viewpoints, and generalizes well to unseen objects.
• Figure 2: Pipeline of UNIC. UNIC integrates four sensing modalities as inputs—point clouds, end-effector rotation, force–torque, and tactile marker displacements—and outputs an extrinsic contact affordance map. We adopt a masked multimodal fusion strategy to ensure a robust multimodal representation learning. In addition, UNIC employs a sampling strategy designed to enhance computational efficiency.
• Figure 3: Illustration of proposed prior-free contact affordance representation over three different cases. For each case, we show the RGB image and the corresponding point cloud. The human operator annotates the contact points. Based on these annotations, the proposed Gaussian kernel-based generation method is applied to produce the contact affordance map. Affordance values range from $-1$ to $1$, with higher values at contact locations and lower values in non-contact regions.
• Figure 4: Illustration of the sampling process. The gray blocks represent flattened sample points obtained from the camera-captured point cloud, while the orange blocks represent the multimodal feature generated by the trunk. Here, $B$ denotes the batch size, $M$ denotes the number of points in point cloud/sample points, and $D$ denotes the dimensionality of the multimodal feature. See Section \ref{['sec:efficient_sampling']} for details.
• Figure 5: Objects used in our experiments. The left set shows seen objects, while the right set shows unseen objects used only during validation.