Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning

TL;DR

Zebra-CoT introduces a large-scale dataset of 182,384 interleaved text–image reasoning traces across 18 domains to train models for native visual CoT. It spans four main categories—scientific reasoning, 2D/3D visual reasoning, and visual logic games—built from real and synthetic data with careful curation to ensure coherent cross-modal traces. Frontier LLMs show weak zero-shot performance on Zebra-CoT, but scaffolding with initial multimodal steps yields substantial gains, and fine-tuning Anole-7B achieves notable improvements across multiple benchmarks, with Bagel-7B able to emit intrinsic visual CoT for RL-ready reasoning. The open-source dataset and models demonstrate Zebra-CoT’s effectiveness in advancing multimodal reasoning and lay groundwork for reinforcement-learning–driven visual CoT capabilities.

Abstract

Humans often use visual aids, for example diagrams or sketches, when solving complex problems. Training multimodal models to do the same, known as Visual Chain of Thought (Visual CoT), is challenging due to: (1) poor off-the-shelf visual CoT performance, which hinders reinforcement learning, and (2) the lack of high-quality visual CoT training data. We introduce $\textbf{Zebra-CoT}$, a diverse large-scale dataset with 182,384 samples, containing logically coherent interleaved text-image reasoning traces. We focus on four categories of tasks where sketching or visual reasoning is especially natural, spanning scientific questions such as geometry, physics, and algorithms; 2D visual reasoning tasks like visual search and jigsaw puzzles; 3D reasoning tasks including 3D multi-hop inference, embodied and robot planning; visual logic problems and strategic games like chess. Fine-tuning the Anole-7B model on the Zebra-CoT training corpus results in an improvement of +12% in our test-set accuracy and yields up to +13% performance gain on standard VLM benchmark evaluations. Fine-tuning Bagel-7B yields a model that generates high-quality interleaved visual reasoning chains, underscoring Zebra-CoT's effectiveness for developing multimodal reasoning abilities. We open-source our dataset and models to support development and evaluation of visual CoT.

Figures (27)

• Figure 1: We curate a large-scale multimodal dataset by sourcing and cleaning raw traces from real-world domains, and generating synthetic examples using templated reasoning filled in by VLMs. Zebra-CoT comprises 4 major categories and 18 subcategories, encompassing over 182K instances in total. A detailed breakdown of the data statistics appears in \ref{['tab:data_statistics']}.
• Figure 2: Visual CoT helps answer complex visual reasoning questions, as illustrated by examples from Zebra-CoT.
• Figure 3: An overview of our data curation pipeline.
• Figure 4: Scaffolding experiment with frontier models. Q represents zero-shot question-only evaluation, 1MT denotes a question with the first multimodal reasoning step provided, and 2MT indicates a question with the first two multimodal reasoning steps. We show that even frontier models with the best multimodal reasoning capabilities perform poorly overall on tasks in Zebra-CoT. However, as we provide the first one or two multimodal steps to those models, the accuracy improves significantly.
• Figure 5: Example interleaved reasoning chains generated by Bagel-Zebra-CoT, a Bagel-7B model finetuned on Zebra-CoT. These traces demonstrate Zebra-CoT's for instilling intrinsic visual reasoning capability in complex multimodal models.