Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The advantages of context specific language models: the case of the Erasmian Language Model

João Gonçalves, Nick Jelicic, Michele Murgia, Evert Stamhuis

TL;DR

The paper addresses the sustainability and governance issues of scaling large language models by proposing a context-specific alternative: the Erasmian Language Model (ELM), a 900M-parameter LLaMA 2-based system trained on Erasmus University Rotterdam data. Through domain-focused pretraining, instruction tuning, and RLHF with direct preference optimization, ELM demonstrates specialization in Erasmus-relevant social sciences, humanities, and medicine tasks while operating on limited hardware and with strong privacy controls. Evaluations include qualitative classroom use, a domain-tailored MMLU assessment, and the COMPASS trustworthiness framework, showing useful performance within its context and highlighting the benefits and trade-offs of context-specific approaches. The study argues that such context-bound models offer practical, governance-aligned alternatives for resource-constrained institutions, providing a replicable methodology for domain-focused AI while emphasizing iterative development and stakeholder engagement.

Abstract

The current trend to improve language model performance seems to be based on scaling up with the number of parameters (e.g. the state of the art GPT4 model has approximately 1.7 trillion parameters) or the amount of training data fed into the model. However this comes at significant costs in terms of computational resources and energy costs that compromise the sustainability of AI solutions, as well as risk relating to privacy and misuse. In this paper we present the Erasmian Language Model (ELM) a small context specific, 900 million parameter model, pre-trained and fine-tuned by and for Erasmus University Rotterdam. We show how the model performs adequately in a classroom context for essay writing, and how it achieves superior performance in subjects that are part of its context. This has implications for a wide range of institutions and organizations, showing that context specific language models may be a viable alternative for resource constrained, privacy sensitive use cases.

The advantages of context specific language models: the case of the Erasmian Language Model

TL;DR

The paper addresses the sustainability and governance issues of scaling large language models by proposing a context-specific alternative: the Erasmian Language Model (ELM), a 900M-parameter LLaMA 2-based system trained on Erasmus University Rotterdam data. Through domain-focused pretraining, instruction tuning, and RLHF with direct preference optimization, ELM demonstrates specialization in Erasmus-relevant social sciences, humanities, and medicine tasks while operating on limited hardware and with strong privacy controls. Evaluations include qualitative classroom use, a domain-tailored MMLU assessment, and the COMPASS trustworthiness framework, showing useful performance within its context and highlighting the benefits and trade-offs of context-specific approaches. The study argues that such context-bound models offer practical, governance-aligned alternatives for resource-constrained institutions, providing a replicable methodology for domain-focused AI while emphasizing iterative development and stakeholder engagement.

Abstract

The current trend to improve language model performance seems to be based on scaling up with the number of parameters (e.g. the state of the art GPT4 model has approximately 1.7 trillion parameters) or the amount of training data fed into the model. However this comes at significant costs in terms of computational resources and energy costs that compromise the sustainability of AI solutions, as well as risk relating to privacy and misuse. In this paper we present the Erasmian Language Model (ELM) a small context specific, 900 million parameter model, pre-trained and fine-tuned by and for Erasmus University Rotterdam. We show how the model performs adequately in a classroom context for essay writing, and how it achieves superior performance in subjects that are part of its context. This has implications for a wide range of institutions and organizations, showing that context specific language models may be a viable alternative for resource constrained, privacy sensitive use cases.
Paper Structure (20 sections, 4 figures, 1 table)

This paper contains 20 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Improving language model efficiency
  4. context-specific language models
  5. Epistemic-functional approach
  6. Methods
  7. Model architecture
  8. Datasets
  9. EUR Research output
  10. EUR Theses
  11. Instruction dataset
  12. Reinforcement learning
  13. Training
  14. Results
  15. Qualitative assessment
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Research papers per year (research output)
  • Figure 2: Number of theses per year
  • Figure 3: Performance of ELM per subject according to MMLU benchmark
  • Figure 4: COMPASS Assessment