Fox-1: Open Small Language Model for Cloud and Edge

TL;DR

Fox-1 presents a 1.6B decoder-only small language model that combines a 3-stage data curriculum with a deeper architecture and a large vocabulary to achieve competitive performance against larger open models. The approach leverages Grouped Query Attention and Rotary Positional Embeddings to boost efficiency and long-context capabilities, while a carefully staged training regime mitigates the cost of long-sequence training. On standard benchmarks, Fox-1 outperforms or matches multiple baselines such as Gemma-2B, Qwen-1.5-1.8B, StableLM-2-1.6B, and OpenELM-1.1B, with strong GSM8k results and robust inference throughput. The authors release the model weights under Apache 2.0 to promote open-source accessibility and broader democratization of LLM technology.

Abstract

We present Fox-1, a series of small language models (SLMs) consisting of Fox-1-1.6B and Fox-1-1.6B-Instruct-v0.1. These models are pre-trained on 3 trillion tokens of web-scraped document data and fine-tuned with 5 billion tokens of instruction-following and multi-turn conversation data. Aiming to improve the pre-training efficiency, Fox-1-1.6B model introduces a novel 3-stage data curriculum across all the training data with 2K-8K sequence length. In architecture design, Fox-1 features a deeper layer structure, an expanded vocabulary, and utilizes Grouped Query Attention (GQA), offering a performant and efficient architecture compared to other SLMs. Fox-1 achieves better or on-par performance in various benchmarks compared to StableLM-2-1.6B, Gemma-2B, Qwen1.5-1.8B, and OpenELM1.1B, with competitive inference speed and throughput. The model weights have been released under the Apache 2.0 license, where we aim to promote the democratization of LLMs and make them fully accessible to the whole open-source community.

Figures (3)

• Figure 1: Fox-1-1.6B compare with other SLMs.
• Figure 2: Performance compared to other SLMs.
• Figure 3: Inference speed compared to other SLMs.