Talking Points: Describing and Localizing Pixels

TL;DR

This work tackles pixel-level grounding of individual keypoints by introducing a bidirectional framework that both describes and localizes pixels in natural language. The Point Descriptor generates free-form descriptions of a keypoint within multi-scale visual context, while the Point Localizer regresses the exact pixel coordinates from those descriptions. A 20K+ image-keypoint-description dataset, LlamaPointInPart, supports supervised training, and cross-category generalization is explored via AP-10K adaptation and reinforcement learning with a frozen localizer as reward. Results show near-ground-truth localization on the new dataset and notable improvements over baselines, with promising cross-category gains when scaling descriptions via RL. The approach offers a path toward scalable, pixel-precise language grounding and may enable future language-guided precise localization across diverse categories and viewpoints.

Abstract

Vision-language models have achieved remarkable success in cross-modal understanding. Yet, these models remain limited to object-level or region-level grounding, lacking the capability for pixel-precise keypoint comprehension through natural language. We introduce a novel framework for pixel level grounding. The framework consists of two complementary components: a Point Descriptor that generates rich, contextual descriptions of individual keypoints, and a Point Localizer that regresses precise pixel coordinates from these descriptions. Unlike prior work that relies on templated prompts or keypoint names, our approach produces free-form, coarse-to-fine descriptions that situate keypoints within their visual context. Since there is no available dataset to train such a system, we introduce LlamaPointInPart, a carefully curated dataset of 20K+ image-keypoint-description triplets synthesized from multiple vision-language models, capturing multi-scale information from scene-level context to visual features around the keypoint. For cross-category generalization, we optimize the Point Descriptor on AP-10K via GRPO, using the frozen Point Localizer as a reward model to produce descriptions that maximize localization accuracy. To evaluate our results we establish a new evaluation protocol. Instead of comparing the text description produced by our method to the ground truth, we use the localizer to determine how close is the predicted point generated to the ground truth point. Experiments demonstrate superior performance compared to baseline models on LlamaPointInPart.The bidirectional nature of our framework should enable future applications in both keypoint-guided image understanding and language-guided precise localization. Our code and dataset are publicly available at https://github.com/matanr/Talking_Points.

Figures (8)

• Figure 1: Talking Points: Describing and Localizing Pixels. Given an image and keypoint (left, red star), we generate rich descriptions (top right) progressing from scene-level object localization, through part identification, to position within part, and local visual features. We evaluate the descriptions by localizing them back to pixels (green point in image). Our TalkingPoints (TP) achieves near GT performance (bottom right), doubling our baseline (OMG-LLaVA) and outperforming ChatGPT-5 and human annotations, which surprisingly perform worst, highlighting the challenge of pixel-level grounding. Evaluation uses mPCK (mean Percentage of Correct Keypoints), measuring the fraction of predictions within a normalized distance threshold of ground truth, averaged across fine and coarse thresholds.
• Figure 1: Performance comparison on LlamaPointInPart test set. Our approach substantially outperforms the OMG-LLaVA baseline, with the Point Descriptor achieving near ground-truth performance.
• Figure 2: LlamaPointInPart dataset construction pipeline: We start with part-annotated datasets, and select the highest-scoring SIFT keypoint within some part (red star). LLaVA generates fine-grained local descriptions from cropped regions around keypoints, while OMG-LLaVA provides object-centric context from full images. Llama3.3 synthesizes these multi-scale perspectives with hierarchical spatial annotations into coherent, coarse-to-fine descriptions, forming our final image-point-description triplets.
• Figure 3: Talking Point Architecture. The Point Descriptor (blue) generates textual descriptions from image-keypoint pairs using Gaussian masks centered at keypoints using different type of token features. See legend at top of figure. The Point Localizer (green) regresses keypoint coordinates from image-description pairs through a special $<$SEG$>$ token that encodes visual information. While the localizer can be used standalone, it can also be applied to the generated descriptions (red dashed arrow) to evaluate localization performance.
• Figure 4: Qualitative keypoints visualization.