Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs

TL;DR

This paper tackles the data-quality bottleneck in fully open multimodal LLMs by introducing Honey-Data-15M, a 15-million-sample high-quality SFT corpus enriched with dual-level Chain-of-Thought reasoning, and a transparent data-curation stack, HoneyPipe/DataStudio. They demonstrate the effectiveness of this approach by training Bee-8B, an 8B model that sets new SOTA performance for fully open MLLMs and rivals recent semi-open models, particularly in complex reasoning and factual accuracy. Through extensive ablations, the authors show that noise reduction and deliberate CoT enrichment are the primary drivers of performance gains, validating a data-centric pathway over mere data quantity. The work also provides a full-stack, open-source pipeline, training recipes, evaluation harness, and model weights to empower the community to reproduce and extend high-quality open MLLMs.

Abstract

Fully open multimodal large language models (MLLMs) currently lag behind proprietary counterparts, primarily due to a significant gap in data quality for supervised fine-tuning (SFT). Existing open-source datasets are often plagued by widespread noise and a critical deficit in complex reasoning data, such as Chain-of-Thought (CoT), which hinders the development of advanced model capabilities. Addressing these challenges, our work makes three primary contributions. First, we introduce Honey-Data-15M, a new SFT dataset comprising approximately 15 million QA pairs, processed through multiple cleaning techniques and enhanced with a novel dual-level (short and long) CoT enrichment strategy. Second, we introduce HoneyPipe, the data curation pipeline, and its underlying framework DataStudio, providing the community with a transparent and adaptable methodology for data curation that moves beyond static dataset releases. Finally, to validate our dataset and pipeline, we train Bee-8B, an 8B model on Honey-Data-15M. Experiments show that Bee-8B establishes a new state-of-the-art (SOTA) for fully open MLLMs, achieving performance that is competitive with, and in some cases surpasses, recent semi-open models such as InternVL3.5-8B. Our work delivers to the community a suite of foundational resources, including: the Honey-Data-15M corpus; the full-stack suite comprising HoneyPipe and DataStudio; training recipes; an evaluation harness; and the model weights. This effort demonstrates that a principled focus on data quality is a key pathway to developing fully open MLLMs that are highly competitive with their semi-open counterparts.

Figures (5)

• Figure 1: Overview of the HoneyPipe. After initial data preparation and filtering, the foundational path generates short CoT responses, which are then checked by Fidelity Verification. Complex instructions that fail this check are routed to the long CoT Enrichment loop. Here, a more powerful model generates a detailed long CoT response, which then undergoes the same Fidelity Verification. This dual-level architecture systematically builds the Honey-Data-15M.
• Figure 2: Distributions of Honey-Data-15M.
• Figure 3: Data collection of Honey-Data-15M. A detailed breakdown of our dataset's composition across seven major categories. The number of samples (in thousands) is listed for each source. The * denotes that the data contains the long CoT response.
• Figure 4: Visual comparison of model performance in our ablation study. The radar chart illustrates the step-by-step impact of our curation pipeline. The performance lift from $D_{\text{raw}}$ (baseline) to $D_{\text{no-CoT}}$ shows the benefit of noise filtering, while the larger leap to $D_{\text{curated}}$ highlights the critical contribution of Chain-of-Thought enrichment, especially in reasoning-heavy domains.
• Figure 5: Performance comparison of models fine-tuned on different 1M data subsets. The model trained on our curated Honey-Data-1M consistently and outperforms both the Qwen2.5-VL-7B base model and the model trained on a Random-1M subset, highlighting the efficiency of our data selection strategy.