Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Study

TL;DR

The paper investigates how well language-model-based code completion works for a functional language, specifically Haskell, by fine-tuning UniXcoder and CodeGPT on Blastwind and evaluating with a manually translated HumanEval-Haskell dataset. It finds that pre-training on imperative languages offers limited transfer to Haskell, while fine-tuning yields meaningful improvements, though performance remains below that seen for imperative languages. Manual analysis reveals distinct model behaviors, with CodeGPT tending toward empty or verbose predictions and UniXcoder producing more incomplete or incorrect outputs, underscoring the need for better datasets and evaluation methods. The work contributes a publicly released Haskell dataset and replication materials, highlighting practical implications for Haskell tooling and guiding future research toward richer FP data and broader model evaluations.

Abstract

Language model-based code completion models have quickly grown in use, helping thousands of developers write code in many different programming languages. However, research on code completion models typically focuses on imperative languages such as Python and JavaScript, which results in a lack of representation for functional programming languages. Consequently, these models often perform poorly on functional languages such as Haskell. To investigate whether this can be alleviated, we evaluate the performance of two language models for code, CodeGPT and UniXcoder, on the functional programming language Haskell. We fine-tune and evaluate the models on Haskell functions sourced from a publicly accessible Haskell dataset on HuggingFace. Additionally, we manually evaluate the models using our novel translated HumanEval dataset. Our automatic evaluation shows that knowledge of imperative programming languages in the pre-training of LLMs may not transfer well to functional languages, but that code completion on functional languages is feasible. Consequently, this shows the need for more high-quality Haskell datasets. A manual evaluation on HumanEval-Haskell indicates CodeGPT frequently generates empty predictions and extra comments, while UniXcoder more often produces incomplete or incorrect predictions. Finally, we release HumanEval-Haskell, along with the fine-tuned models and all code required to reproduce our experiments on GitHub (https://github.com/AISE-TUDelft/HaskellCCEval).

Figures (3)

• Figure 1: Approach pipeline.
• Figure 2: Blastwind sample.
• Figure 3: HumanEval-Haskell sample.