Falcon2-11B Technical Report

TL;DR

Falcon2-11B introduces a scalable 11B LLM with a four-stage pretraining regime designed for long-context, multilingual, and code-capable performance, paired with Falcon2-11B-VLM for vision-language tasks. The architecture leverages GQA multiquery with $ ext{n}_ ext{kv}=8$, FA2 acceleration, and stage-based context-length growth, plus language-tuned multilingual data filters and a mix of RefinedWeb and curated corpora. Empirical results show strong English and multilingual benchmarks, competitive code-generation scores, and favorable multimodal benchmark averages relative to open-source peers of similar size; the VLM demonstrates solid cross-benchmark performance using a frozen CLIP visual encoder with a two-stage training regime. The work also discusses stability strategies (spike management, batch-size ramping) and releases both weights and code under a permissive license, facilitating open adoption and downstream finetuning across multilingual and multimodal applications.

Abstract

We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model. We report our findings during the training of the Falcon2-11B which follows a multi-stage approach where the early stages are distinguished by their context length and a final stage where we use a curated, high-quality dataset. Additionally, we report the effect of doubling the batch size mid-training and how training loss spikes are affected by the learning rate. The downstream performance of the foundation model is evaluated on established benchmarks, including multilingual and code datasets. The foundation model shows strong generalization across all the tasks which makes it suitable for downstream finetuning use cases. For the vision language model, we report the performance on several benchmarks and show that our model achieves a higher average score compared to open-source models of similar size. The model weights and code of both Falcon2-11B and Falcon2-11B-vlm are made available under a permissive license.

Figures (4)

• Figure 1: Visualization of conversation tree flattening and loss masking. In this simple example, the token count of the conversation tree is 19, while the flattened conversation tree contains the 19 initial tokens, but also contains 13 repeated tokens with masked loss, producing an overhead of $+68\%$.
• Figure 2: Stage 1 training: cross-entropy loss of next token prediction in logarithm (left) and linear (right) scale of trained tokens. The right plot also shows a cumulative count of spikes which caused a rollback and restart of training. Each of the 4 stars depicts a doubling of batch size, resulting in a history of $2K \rightarrow 4K \rightarrow 8K \rightarrow 16K \rightarrow 32K$ of the number of samples in the batch. Presumably, batch size doublings helped to reduce the (initially large) frequency of spikes. Finally, the diamond depicts the change of Adam $\varepsilon$ parameter from $10^{-8}$ to $10^{-7}$.
• Figure 3: Evaluation results across stages 1-3. We present the progression of evaluation results on English NLP tasks across different training checkpoints in stages 1-3. Final results from the following models are given as reference: Falcon-7B, Falcon-40B, Llama2-7B, Llama2-13B, Llama-30B, Qwen-7B, and Yi-6B.
• Figure 4: Falcon2-11B VLM architecture. See \ref{['falcon_vlm']} for details.