Rethinking Visual Intelligence: Insights from Video Pretraining

TL;DR

This work investigates whether pretraining on spatiotemporal data can endow vision systems with inductive priors that enable rapid, data-efficient problem solving across diverse visually grounded tasks. It introduces a unified framework to adapt Video Diffusion Models (VDMs) to image-to-image tasks by reframing examples as short transition videos and compares them to pretrained Large Language Models (LLMs) using LoRA adapters, under symmetric evaluation. Across ARC-AGI, ConceptARC, visual games, route planning, and cellular automata, VDMs demonstrate higher data efficiency and better generalization in many tasks, highlighting the value of modality-aligned pretraining for visual intelligence. The findings suggest a promising path toward visual foundation models that combine generative and structured problem-solving capabilities, with potential implications for planning, simulation, and robotics in embodied settings.

Abstract

Large language models (LLMs) have demonstrated that large-scale pretraining enables systems to adapt rapidly to new problems with little supervision in the language domain. This success, however, has not translated as effectively to the visual domain, where models, including LLMs, continue to struggle with compositional understanding, sample efficiency, and general-purpose problem-solving. We investigate Video Diffusion Models (VDMs) as a promising direction for bridging this gap. Pretraining on spatiotemporal data endows these models with strong inductive biases for structure and dynamics, which we hypothesize can support broad task adaptability. To test this, we design a controlled evaluation in which both a pretrained LLM and a pretrained VDM are equipped with lightweight adapters and presented with tasks in their natural modalities. Across benchmarks including ARC-AGI, ConceptARC, visual games, route planning, and cellular automata, VDMs demonstrate higher data efficiency than their language counterparts. Taken together, our results indicate that video pretraining offers inductive biases that support progress toward visual foundation models.

Figures (39)

• Figure 1: Radar plot showing ConceptARC competencies between VDMs and LLMs, GPT-4 [IC] is added for additional reference.
• Figure 1: ARC-AGI test performance. Following the official evaluation protocol chollet2024arc, models are evaluated with two attempts per test input. We also report single-attempt results for comparability with commercial LLMs, which are only available under this setting.
• Figure 2: Venn diagram of ARC-AGI tasks showing those solved exclusively by each model and those solved by both.
• Figure 3: Qualitative results on ARC-AGI for problems 0607ce86, 7ee1c6ea, and f45f5ca7.
• Figure 4: Accuracy as a function of training set size for CogVideoX1.5-5B and Qwen3-4B-Instruct-2507 on five visual games.