Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HoneyBee: Data Recipes for Vision-Language Reasoners

Hritik Bansal, Devandra Singh Sachan, Kai-Wei Chang, Aditya Grover, Gargi Ghosh, Wen-tau Yih, Ramakanth Pasunuru

TL;DR

The paper tackles how to construct effective vision-language reasoning datasets and presents HoneyBee, a large-scale CoT dataset with $2.5$M examples built from diverse contexts, interventions, and scaling strategies. It systematically evaluates context sources, a suite of perception and problem-solving interventions, and scaling along image, question, and CoT axes, revealing which choices yield robust performance gains. HoneyBee-trained VL reasoners outperform state-of-the-art baselines across model sizes, and the authors demonstrate additional benefits from test-time decoding strategies that drastically reduce token usage. The work provides a principled, scalable data curation recipe for advancing VL reasoning and highlights the importance of data design choices alongside model scale. Practically, these insights enable more efficient, generalizable multimodal reasoning systems and set a framework for future data-centric VL research.

Abstract

Recent advances in vision-language models (VLMs) have made them highly effective at reasoning tasks. However, the principles underlying the construction of performant VL reasoning training datasets remain poorly understood. In this work, we introduce several data curation approaches and study their impacts on VL reasoning capabilities by carefully controlling training and evaluation setups. We analyze the effects of context (image and question pair) sources, implement targeted data interventions, and explore scaling up images, questions, and chain-of-thought (CoT) solutions. Our findings reveal that (a) context source strategies significantly affect VLM performance, (b) interventions such as auxiliary signals from image captions and the inclusion of text-only reasoning yield substantial gains, and (c) scaling all data dimensions (e.g., unique questions per image and unique CoTs per image-question pair) consistently improves reasoning capability. Motivated by these insights, we introduce HoneyBee, a large-scale, high-quality CoT reasoning dataset with 2.5M examples consisting 350K image-question pairs. VLMs trained with HoneyBee outperform state-of-the-art models across model sizes. For instance, a HoneyBee-trained VLM with 3B parameters outperforms the SOTA model and the base model by 7.8% and 24.8%, respectively, on MathVerse. Furthermore, we propose a test-time scaling strategy that reduces decoding cost by 73% without sacrificing accuracy. Overall, this work presents improved strategies for VL reasoning dataset curation research.

HoneyBee: Data Recipes for Vision-Language Reasoners

TL;DR

The paper tackles how to construct effective vision-language reasoning datasets and presents HoneyBee, a large-scale CoT dataset with M examples built from diverse contexts, interventions, and scaling strategies. It systematically evaluates context sources, a suite of perception and problem-solving interventions, and scaling along image, question, and CoT axes, revealing which choices yield robust performance gains. HoneyBee-trained VL reasoners outperform state-of-the-art baselines across model sizes, and the authors demonstrate additional benefits from test-time decoding strategies that drastically reduce token usage. The work provides a principled, scalable data curation recipe for advancing VL reasoning and highlights the importance of data design choices alongside model scale. Practically, these insights enable more efficient, generalizable multimodal reasoning systems and set a framework for future data-centric VL research.

Abstract

Recent advances in vision-language models (VLMs) have made them highly effective at reasoning tasks. However, the principles underlying the construction of performant VL reasoning training datasets remain poorly understood. In this work, we introduce several data curation approaches and study their impacts on VL reasoning capabilities by carefully controlling training and evaluation setups. We analyze the effects of context (image and question pair) sources, implement targeted data interventions, and explore scaling up images, questions, and chain-of-thought (CoT) solutions. Our findings reveal that (a) context source strategies significantly affect VLM performance, (b) interventions such as auxiliary signals from image captions and the inclusion of text-only reasoning yield substantial gains, and (c) scaling all data dimensions (e.g., unique questions per image and unique CoTs per image-question pair) consistently improves reasoning capability. Motivated by these insights, we introduce HoneyBee, a large-scale, high-quality CoT reasoning dataset with 2.5M examples consisting 350K image-question pairs. VLMs trained with HoneyBee outperform state-of-the-art models across model sizes. For instance, a HoneyBee-trained VLM with 3B parameters outperforms the SOTA model and the base model by 7.8% and 24.8%, respectively, on MathVerse. Furthermore, we propose a test-time scaling strategy that reduces decoding cost by 73% without sacrificing accuracy. Overall, this work presents improved strategies for VL reasoning dataset curation research.
Paper Structure (65 sections, 5 equations, 14 figures, 15 tables)

This paper contains 65 sections, 5 equations, 14 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Data Curation Pipeline
  4. Context Curation
  5. Sourcing.
  6. Mixing.
  7. Data Interventions
  8. Perception Enhancement
  9. Visual Perturbation.
  10. Text-Rich Images.
  11. Perceptual Redundancy.
  12. Shallow Perception.
  13. Caption and Solve.
  14. Problem-Solving Skill Enhancement
  15. Text-Only Reasoning.
  16. ...and 50 more sections

Figures (14)

  • Figure 1: Summary of the results. (a) We show that training with increasing amounts of our curated HoneyBee data leads to consistent accuracy improvements averaged across five VL reasoning tasks (MathVista, MathVerse, MathVision, MMMU-Pro, and We-Math) for several model sizes ($1$B to $8$B). (b) We report the relative gains achieved by training HoneyBee with PLM-3B compared to existing VL reasoning CoT datasets containing 250K–300K VL instances.
  • Figure 2: Overview of the data curation pipeline. In this work, we curate the context (image, question) from diverse sources and assess the impact of their mixing. Further, we curate a set of data interventions that target diverse skills and types. Subsequently, we study the impact of scaling along different data axes. These insights lead to the creation of our large-scale HoneyBee dataset.
  • Figure 3: Overview of data intervention strategies. In this work, we curate a diverse set of data interventions to enhance the quality of VL reasoning CoT data. These methods target a range of skills, such as perception and problem-solving capabilities. We present details for each intervention in § \ref{['sec:data_intervention_explain']}.
  • Figure 4: Impact of scaling diverse data axes in VL reasoning data. We train PLM-3B and PLM-8B on datasets of varying sizes (setup explained in § \ref{['sec:scaling_diverse_modalities']}). The results show that the reasoning performance consistently improves as we scale each data axis: (a) images, (b) synthetic questions per image, and (c) CoTs per (image, question) pair.
  • Figure 5: HoneyBee statistics. (a) We present key statistics of the HoneyBee dataset. Notably, HoneyBee is a large-scale dataset with $2.5$M instances with $350$K unique questions. (b) We also provide statistics on several keywords in the HoneyBee CoTs that elicit diverse reasoning behaviors, such as perceptual understanding, question comprehension, planning, and reflection.
  • ...and 9 more figures