PoseScript: Linking 3D Human Poses and Natural Language

TL;DR

PoseScript tackles the lack of fine-grained natural language descriptions for 3D human poses by introducing PoseScript-H and a scalable PoseScript-A. A modular automatic captioning pipeline uses posecodes to convert 3D keypoints into natural-language descriptions, enabling large-scale pretraining (≈100k poses with captions) for three multi-modal tasks: text-to-pose retrieval, text-conditioned pose generation, and pose description generation. Pretraining on PoseScript-A substantially boosts performance across tasks, with transformer-based text encoders and data augmentation further enhancing results. The dataset and methods advance pose-language understanding, with potential applications in annotation, pose-based search, and animation, and highlight future opportunities in multi-person scenarios and integration with large multimodal models.

Abstract

Natural language plays a critical role in many computer vision applications, such as image captioning, visual question answering, and cross-modal retrieval, to provide fine-grained semantic information. Unfortunately, while human pose is key to human understanding, current 3D human pose datasets lack detailed language descriptions. To address this issue, we have introduced the PoseScript dataset. This dataset pairs more than six thousand 3D human poses from AMASS with rich human-annotated descriptions of the body parts and their spatial relationships. Additionally, to increase the size of the dataset to a scale that is compatible with data-hungry learning algorithms, we have proposed an elaborate captioning process that generates automatic synthetic descriptions in natural language from given 3D keypoints. This process extracts low-level pose information, known as "posecodes", using a set of simple but generic rules on the 3D keypoints. These posecodes are then combined into higher level textual descriptions using syntactic rules. With automatic annotations, the amount of available data significantly scales up (100k), making it possible to effectively pretrain deep models for finetuning on human captions. To showcase the potential of annotated poses, we present three multi-modal learning tasks that utilize the PoseScript dataset. Firstly, we develop a pipeline that maps 3D poses and textual descriptions into a joint embedding space, allowing for cross-modal retrieval of relevant poses from large-scale datasets. Secondly, we establish a baseline for a text-conditioned model generating 3D poses. Thirdly, we present a learned process for generating pose descriptions. These applications demonstrate the versatility and usefulness of annotated poses in various tasks and pave the way for future research in the field.

Figures (22)

• Figure 1: Illustration of three multi-modal learning applications that can be implemented using PoseScript. The top figure illustrates text-to-pose retrieval where the goal is to retrieve poses in a large-scale database given a text query. This can be applied to databases of images with associated SMPL fits. The middle figure shows an example of text-conditioned pose generation. The bottom figure presents the generation of pose descriptions.
• Figure 2: Examples of pose descriptions from PoseScript, produced by human annotators (left) and by our automatic captioning pipeline (right).
• Figure 3: Origin of the selected poses. The top bar plot shows the proportion of sequences that are eventually used in PoseScript with respect to available sequences in AMASS. A sequence is 'used' if it provided at least one pose to PoseScript. The bottom bar plot shows the distribution of the PoseScript poses over the AMASS sub-datasets.
• Figure 4: Interface presented to the AMT annotators in order to collect discriminative descriptions of the blue pose following a two-step process.
• Figure 5: Overview of our captioning pipeline. Given a normalized 3D pose, we use posecodes to extract semantic pose information. These posecodes are then selected, merged or combined (when relevant) before being converted into a structural pose description in natural language. Letters 'L' and 'R' stand for 'left' and 'right' respectively.