Perish or Flourish? A Holistic Evaluation of Large Language Models for Code Generation in Functional Programming

TL;DR

This work tackles the challenge of evaluating large language models for code generation in functional programming by introducing FPEval, a holistic framework built on FPBench that encompasses 721 tasks across Haskell, OCaml, and Scala. The study assesses GPT-3.5, GPT-4o, and GPT-5 (with Java as an imperative baseline) using rigorous test suites and static-analysis-based style checks, revealing substantial gains with model advancement but persistent gaps in pure FP languages and notable non-idiomatic, imperative-like patterns. It also demonstrates that LLMs can partially self-repair when given static-analysis feedback and explicit repair instructions, particularly via instruction-guided repair, which improves code quality and maintainability. The findings highlight a paradigm gap and motivate FP-aware training and evaluation for LLMs, establishing FPBench/FPEval as a foundational benchmark to guide future research and practice in FP-oriented AI-assisted software engineering.

Abstract

Functional programming provides strong foundations for developing reliable and secure software systems, yet its adoption remains not widespread due to the steep learning curve. Recent advances in Large Language Models (LLMs) for code generation present new opportunities to lower these barriers. However, extensive evaluations of LLMs largely focus on imperative programming languages, and their capabilities in functional programming languages (FP) remain underexplored. To address this gap, we introduce FPEval, a holistic evaluation framework built on FPBench, a new benchmark of 721 programming tasks across three difficulty levels on three mainstream FP languages: Haskell, Ocaml and Scala. FPEval provides compehensive evaluation infrastructures with both test validations with comprehensive test suites and static analysis tools to assess both functional correctness and code style and maintainability. Using this framework, we evaluate state-of-the-art LLMs, including GPT-3.5, GPT-4o, and GPT-5, for code generation in functional programming languages and Java as an imperative baseline. Our results demonstrate that LLM performance in functional programming improves substantially with model advancement; however, error rates remain significantly higher in purely functional languages (Haskell and OCaml) than in hybrid (Scala) or imperative (Java) languages. Moreover, LLMs frequently generate non-idiomatic functional code that follows imperative patterns, raising concerns about code style and long-term maintainability. Finally, we show that LLMs can partially self-repair both correctness and quality issues when provided with static analysis feedback and hand-crafted instructions for common types of issues.

Figures (2)

• Figure 1: Task distribution across time
• Figure 2: Pass@1 performance of GPT-4o