Adept: Annotation-Denoising Auxiliary Tasks with Discrete Cosine Transform Map and Keypoint for Human-Centric Pretraining

TL;DR

Adept tackles the data-scarcity problem in human-centric pretraining by discarding depth data and extracting semantic cues from RGB images in the frequency domain using Discrete Cosine Transform maps. It introduces annotation-denoising auxiliary tasks that recover DCT maps and keypoint annotations from noisy latent features with RGB guidance, coupled with a MoCo-style contrastive pretraining objective. Empirical results across pose estimation, human parsing, crowd counting, crowd localization, and person ReID on COCO, MPII, SHA/SHB, and Market1501/MSMT demonstrate consistent improvements over state-of-the-art methods, highlighting the value of low-frequency semantic information and local feature learning. The approach enables scalable, depth-free pretraining with strong cross-task transfer, though it notes biases toward local tasks and substantial compute requirements for broader generalization.

Abstract

Human-centric perception is the core of diverse computer vision tasks and has been a long-standing research focus. However, previous research studied these human-centric tasks individually, whose performance is largely limited to the size of the public task-specific datasets. Recent human-centric methods leverage the additional modalities, e.g., depth, to learn fine-grained semantic information, which limits the benefit of pretraining models due to their sensitivity to camera views and the scarcity of RGB-D data on the Internet. This paper improves the data scalability of human-centric pretraining methods by discarding depth information and exploring semantic information of RGB images in the frequency space by Discrete Cosine Transform (DCT). We further propose new annotation denoising auxiliary tasks with keypoints and DCT maps to enforce the RGB image extractor to learn fine-grained semantic information of human bodies. Our extensive experiments show that when pretrained on large-scale datasets (COCO and AIC datasets) without depth annotation, our model achieves better performance than state-of-the-art methods by +0.5 mAP on COCO, +1.4 PCKh on MPII and -0.51 EPE on Human3.6M for pose estimation, by +4.50 mIoU on Human3.6M for human parsing, by -3.14 MAE on SHA and -0.07 MAE on SHB for crowd counting, by +1.1 F1 score on SHA and +0.8 F1 score on SHA for crowd localization, and by +0.1 mAP on Market1501 and +0.8 mAP on MSMT for person ReID. We also validate the effectiveness of our method on MPII+NTURGBD datasets

Figures (8)

• Figure 1: Comparison of fine-grained semantic information in DCT maps, real depth maps captured by RGB-D cameras, and pseudo-depth maps labeled by ranftl2021vision. DCT map can be generated from images with any external sensors or pretrainied models and contains more fine-grained semantic information. The real depth maps can also show fine-grained information, to some extent. The pseudo-depth maps can be easily generated by external models but it contains little fine-grained semantic information.
• Figure 2: (a) Overview of our proposed Adept framework. Adept first utilizes the Discrete Cosine Transform maps and human key points priors for human-centric pre-training. In the data input, we use dashed lines to represent annotation-denoising auxiliary tasks. Adept then applies the pre-trained model as the backbone for various human-centric perception tasks, e.g., Human Parsing, Pose Estimation, Crowd Counting, Crowd Localization and Person Re-identification. (b) Comparison of ImageNet pretraining and different pretraining methods. Our proposed method improves the performance of various human-centric tasks by a considerable margin.
• Figure 3: (a) Schematic illustration of our proposed Adept method: Learning local and global features. Given a seed image and keypoint, we first generate the DCT map by Discrete Cosine Transform and two image views. Modality-specific encoders extract latent features of each modality. Random noises are added to the latent features of DCT and keypoint, and then task-specific decoders are employed to reconstruct DCT and keypoint by cross-attention with the help of image features, so as to force image encoder to learn fine-grained features. Additionally, contrastive learning is applied between two image views to extract global features. (b) The schematic illustration of transformer decoders for both keypoints and DCT maps. Specifically, we feed the noisy DCT latent representations and keypoint representations in the multi-head self-attention layer and the latent features of images for cross-attention layer.
• Figure 4: The generation pipeline of Discrete Cosine Transform maps. RGB image is first transformed to the YCbCr color space and then divided to patches for each channel. Discrete Cosine Transform operates on each part to get DCT coefficients, which are flattened to 1D vector. DCT maps are generated by concatenating DCT coefficients of all channels.
• Figure 5: Visualization of heat maps for Y, Cb, and Cr components on the COCO validation dataset.