Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding

TL;DR

Molmo2 tackles the limited openness of leading video-language models by delivering a fully open-weight, fully open-data VLM capable of grounding in both images and videos. It introduces nine datasets (dense video captions, long-form QA, video pointing and tracking, and multi-image tasks) and a three-stage training pipeline (image-captioning pre-training, joint supervised fine-tuning, and long-context fine-tuning) augmented by packing, token weighting, and a message-tree encoding scheme. Empirically, Molmo2 achieves state-of-the-art performance among open models on short-video understanding, counting, captioning, and video grounding, while remaining competitive on longer videos and surpassing several proprietary systems on certain grounding tasks. The work emphasizes open science by releasing data, weights, and training code, and demonstrates that open pipelines can reach leading performance in grounding-rich vision-language tasks with broad applicability to search, robotics, and interactive AI systems.

Abstract

Today's strongest video-language models (VLMs) remain proprietary. The strongest open-weight models either rely on synthetic data from proprietary VLMs, effectively distilling from them, or do not disclose their training data or recipe. As a result, the open-source community lacks the foundations needed to improve on the state-of-the-art video (and image) language models. Crucially, many downstream applications require more than just high-level video understanding; they require grounding -- either by pointing or by tracking in pixels. Even proprietary models lack this capability. We present Molmo2, a new family of VLMs that are state-of-the-art among open-source models and demonstrate exceptional new capabilities in point-driven grounding in single image, multi-image, and video tasks. Our key contribution is a collection of 7 new video datasets and 2 multi-image datasets, including a dataset of highly detailed video captions for pre-training, a free-form video Q&A dataset for fine-tuning, a new object tracking dataset with complex queries, and an innovative new video pointing dataset, all collected without the use of closed VLMs. We also present a training recipe for this data utilizing an efficient packing and message-tree encoding scheme, and show bi-directional attention on vision tokens and a novel token-weight strategy improves performance. Our best-in-class 8B model outperforms others in the class of open weight and data models on short videos, counting, and captioning, and is competitive on long-videos. On video-grounding Molmo2 significantly outperforms existing open-weight models like Qwen3-VL (35.5 vs 29.6 accuracy on video counting) and surpasses proprietary models like Gemini 3 Pro on some tasks (38.4 vs 20.0 F1 on video pointing and 56.2 vs 41.1 J&F on video tracking).

Figures (35)

• Figure 1: Molmo2 is trained on one of the largest fully open video-centric multimodal corpus to date, including nine new datasets for dense video captioning, long-form and long-video QA, and open-vocabulary pointing and tracking over images, multi-images, and videos. Molmo2 accepts single images, image sets, and videos as input and can produce both free-form language and grounded outputs such as spatio-temporal points, object tracks, and grounded chain-of-thoughts that localize objects and events over time. Across diverse video-language and grounding benchmarks, Molmo2 matches or surpasses prior open models, approaches proprietary systems, and remains fully open.
• Figure 2: Molmo2 follows the standard design of connecting a vision encoder and a language model to process video inputs.
• Figure 3: Attention mask for a packed sequence with two examples. The first contains two QA pairs for one image. Frame tokens (dark pink) have forward attention, while masking blocks cross-attention between different examples (lower-left empty block) and between distinct QA pairs within the same example (upper empty block).
• Figure 4: Molmo2 SFT mixture. Categories and datasets are shown in proportion to sampling rates in SFT mixture.
• Figure 5: Elo ratings with confidence intervals