Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MojoBench: Language Modeling and Benchmarks for Mojo

Nishat Raihan, Joanna C. S. Santos, Marcos Zampieri

TL;DR

MojoBench is introduced, the first framework for Mojo code generation, and Mojo-Coder, the first LLM pretrained and finetuned for Mojo code generation, which supports instructions in 5 natural languages (NLs).

Abstract

The recently introduced Mojo programming language (PL) by Modular, has received significant attention in the scientific community due to its claimed significant speed boost over Python. Despite advancements in code Large Language Models (LLMs) across various PLs, Mojo remains unexplored in this context. To address this gap, we introduce MojoBench, the first framework for Mojo code generation. MojoBench includes HumanEval-Mojo, a benchmark dataset designed for evaluating code LLMs on Mojo, and Mojo-Coder, the first LLM pretrained and finetuned for Mojo code generation, which supports instructions in 5 natural languages (NLs). Our results show that Mojo-Coder achieves a 30-35% performance improvement over leading models like GPT-4o and Claude-3.5-Sonnet. Furthermore, we provide insights into LLM behavior with underrepresented and unseen PLs, offering potential strategies for enhancing model adaptability. MojoBench contributes to our understanding of LLM capabilities and limitations in emerging programming paradigms fostering more robust code generation systems.

MojoBench: Language Modeling and Benchmarks for Mojo

TL;DR

MojoBench is introduced, the first framework for Mojo code generation, and Mojo-Coder, the first LLM pretrained and finetuned for Mojo code generation, which supports instructions in 5 natural languages (NLs).

Abstract

The recently introduced Mojo programming language (PL) by Modular, has received significant attention in the scientific community due to its claimed significant speed boost over Python. Despite advancements in code Large Language Models (LLMs) across various PLs, Mojo remains unexplored in this context. To address this gap, we introduce MojoBench, the first framework for Mojo code generation. MojoBench includes HumanEval-Mojo, a benchmark dataset designed for evaluating code LLMs on Mojo, and Mojo-Coder, the first LLM pretrained and finetuned for Mojo code generation, which supports instructions in 5 natural languages (NLs). Our results show that Mojo-Coder achieves a 30-35% performance improvement over leading models like GPT-4o and Claude-3.5-Sonnet. Furthermore, we provide insights into LLM behavior with underrepresented and unseen PLs, offering potential strategies for enhancing model adaptability. MojoBench contributes to our understanding of LLM capabilities and limitations in emerging programming paradigms fostering more robust code generation systems.

Paper Structure

This paper contains 49 sections, 16 equations, 8 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background & Motivation
  3. Related Work
  4. PL-specific Corpora
  5. PL-specific Benchmarks
  6. PL-specific LLMs
  7. MojoBench
  8. Mojo-Corpus: A Corpus of Mojo Code
  9. Mojo-SFT & Mojo-mSFT
  10. Mojo-SFT
  11. Mojo-mSFT
  12. HumanEval-Mojo
  13. Mojo-Coder
  14. Base Model Selection
  15. Pretraining
  16. ...and 34 more sections

Figures (8)

  • Figure 1: The complete workflow of developing MojoBench and all of its components. See Section \ref{['sec:humaneval_mojo']} for HumanEval-Mojo.
  • Figure 2: The workflow of compiling Mojo-mSFT from Mojo-SFT. Similar to the approach adopted by raihan2024mHumanEval.
  • Figure 3: Code Lengths in both Mojo-SFT & Mojo-mSFT.
  • Figure 4: A sample prompt from HumanEval.
  • Figure 5: A sample prompt from HumanEval-Mojo.
  • ...and 3 more figures