HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild

TL;DR

This work tackles the open-world problem of identifying, segmenting, and tracking hand-held objects in unconstrained videos. It introduces HOIST-Former, a transformer-based architecture that uses a novel Hand-Object Transformer Decoder to jointly reason about hands and their held objects via mutual feature pooling, guided by a contact-loss signal that emphasizes hand-object interactions. A new in-the-wild HOIST dataset (4,228 videos, ~83,970 frames) provides rich annotations (bounding boxes, masks, tracking IDs) to train and evaluate the method, with masks generated for training through Segment Anything on annotated boxes. Experimental results demonstrate that HOIST-Former outperforms a broad set of baselines, including Mask2Former variants and various tracking pipelines, across HOIST and cross-dataset evaluations, highlighting the importance of bidirectional hand–object context and contact-aware supervision for robust hand-held object segmentation and tracking.

Abstract

We address the challenging task of identifying, segmenting, and tracking hand-held objects, which is crucial for applications such as human action segmentation and performance evaluation. This task is particularly challenging due to heavy occlusion, rapid motion, and the transitory nature of objects being hand-held, where an object may be held, released, and subsequently picked up again. To tackle these challenges, we have developed a novel transformer-based architecture called HOIST-Former. HOIST-Former is adept at spatially and temporally segmenting hands and objects by iteratively pooling features from each other, ensuring that the processes of identification, segmentation, and tracking of hand-held objects depend on the hands' positions and their contextual appearance. We further refine HOIST-Former with a contact loss that focuses on areas where hands are in contact with objects. Moreover, we also contribute an in-the-wild video dataset called HOIST, which comprises 4,125 videos complete with bounding boxes, segmentation masks, and tracking IDs for hand-held objects. Through experiments on the HOIST dataset and two additional public datasets, we demonstrate the efficacy of HOIST-Former in segmenting and tracking hand-held objects.

Figures (5)

• Figure 1: HOIST-Former consists of a backbone network, a pixel decoder, and a transformer decoder. The input video is initially processed through the backbone network and the pixel decoder to generate high-resolution spatio-temporal features $\mathcal{F}$. The transformer decoder operates on $\mathcal{F}$, decoding a set of N hand queries and their corresponding object queries, resulting in N spatio-temporal hand masks and corresponding object masks.
• Figure 2: The Hand-Object Transformer Decoder features a network architecture with $L$ layers. This figure demonstrates the operational flow of a single layer, which includes two mask attention modules and two cross-attention modules. The inputs for this layer consist of N sets of four elements each: a hand query, an object query, a spatio-temporal hand mask, and a spatio-temporal object mask. The outputs of this layer are the correspondingly updated versions of these entities.
• Figure 3: Sample frames from HOIST. HOIST dataset contains videos with diverse scenes, camera views, object sizes, and occlusions.
• Figure 4: Illustrative qualitative results of HOIST-Former. Each row displays selected frames from a single video. Within each row, a distinct hand-held object is assigned a unique tracking ID and is consistently represented in the same color.
• Figure 5: Some failure cases of HOIST-Former. The first row corresponds to two frames from different videos. The last row corresponds to two frames from the same video.