Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HARE: HumAn pRiors, a key to small language model Efficiency

Lingyun Zhang, Bin jin, Gaojian Ge, Lunhui Liu, Xuewen Shen, Mingyong Wu, Houqian Zhang, Yongneng Jiang, Shiqi Chen, Shi Pu

TL;DR

This work addresses the inefficiency of small language models in resource-constrained settings by injecting human priors into data construction. It proposes a principled pipeline that combines semantic-diverse, high-quality open data ($D_1$) with synthesised data ($D_2$) and NLP-task data ($D_3$), followed by strict decontamination to prevent benchmark leakage. A 1.1B parameter model, HARE-1.1B, trained on this dataset, demonstrates favorable performance on Open LLM Leaderboard benchmarks and shows reduced data-leakage risk compared with several baselines. The study highlights the practical viability and benefits of incorporating human priors in data construction for efficient SLM training, while noting limitations and avenues for future work in priors quality and resource-aware design.

Abstract

Human priors play a crucial role in efficiently utilizing data in deep learning. However, with the development of large language models (LLMs), there is an increasing emphasis on scaling both model size and data volume, which often diminishes the importance of human priors in data construction. Influenced by these trends, existing Small Language Models (SLMs) mainly rely on web-scraped large-scale training data, neglecting the proper incorporation of human priors. This oversight limits the training efficiency of language models in resource-constrained settings. In this paper, we propose a principle to leverage human priors for data construction. This principle emphasizes achieving high-performance SLMs by training on a concise dataset that accommodates both semantic diversity and data quality consistency, while avoiding benchmark data leakage. Following this principle, we train an SLM named HARE-1.1B. Extensive experiments on large-scale benchmark datasets demonstrate that HARE-1.1B performs favorably against state-of-the-art SLMs, validating the effectiveness of the proposed principle. Additionally, this provides new insights into efficient language model training in resource-constrained environments from the view of human priors.

HARE: HumAn pRiors, a key to small language model Efficiency

TL;DR

This work addresses the inefficiency of small language models in resource-constrained settings by injecting human priors into data construction. It proposes a principled pipeline that combines semantic-diverse, high-quality open data () with synthesised data () and NLP-task data (), followed by strict decontamination to prevent benchmark leakage. A 1.1B parameter model, HARE-1.1B, trained on this dataset, demonstrates favorable performance on Open LLM Leaderboard benchmarks and shows reduced data-leakage risk compared with several baselines. The study highlights the practical viability and benefits of incorporating human priors in data construction for efficient SLM training, while noting limitations and avenues for future work in priors quality and resource-aware design.

Abstract

Human priors play a crucial role in efficiently utilizing data in deep learning. However, with the development of large language models (LLMs), there is an increasing emphasis on scaling both model size and data volume, which often diminishes the importance of human priors in data construction. Influenced by these trends, existing Small Language Models (SLMs) mainly rely on web-scraped large-scale training data, neglecting the proper incorporation of human priors. This oversight limits the training efficiency of language models in resource-constrained settings. In this paper, we propose a principle to leverage human priors for data construction. This principle emphasizes achieving high-performance SLMs by training on a concise dataset that accommodates both semantic diversity and data quality consistency, while avoiding benchmark data leakage. Following this principle, we train an SLM named HARE-1.1B. Extensive experiments on large-scale benchmark datasets demonstrate that HARE-1.1B performs favorably against state-of-the-art SLMs, validating the effectiveness of the proposed principle. Additionally, this provides new insights into efficient language model training in resource-constrained environments from the view of human priors.
Paper Structure (16 sections, 3 figures, 8 tables)

This paper contains 16 sections, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Data Construction
  3. Training
  4. Experiments
  5. Ablation studies
  6. Overall performance
  7. Conclusion
  8. Data Construction
  9. Collected open-source data
  10. Data synthesis using LLMs
  11. Mixture of synthetic and open-source NLP task data
  12. Data decontamination process
  13. Data Leakage
  14. Supervised Fine-tuning
  15. Chat
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Illustration showing the performances of representative SLMs zhang2024tinyllamamehta2024openelmli2023textbooks on the benchmark datasets in the open LLM leaderboard open-llm-leaderboard at different levels of human prior knowledge. As the integration of human priors deepens, models demonstrate improved performance, but with an increased risk of data leakage.
  • Figure 2: Data Construction Pipeline and Composition. $D_1$: high-quality categorized open-source cleaned data. $D_2$: semantic enriched synthetic data. $D_3$: mixture of synthetic and open-source NLP task data.
  • Figure 3: Ablation studies showing the average score curves of 0.25B models on the Open LLM Leaderboard as the number of training tokens increases.