Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MLCPD: A Unified Multi-Language Code Parsing Dataset with Universal AST Schema

Jugal Gajjar, Kamalasankari Subramaniakuppusamy

TL;DR

MLCPD addresses cross-language code understanding by providing a unified, lossless AST-based representation across ten languages. It constructs over seven million parsed files with a four-layer universal schema and Tree-sitter-driven parsing, enabling scalable, language-agnostic analyses. The authors demonstrate cross-language structural regularities and robust mapping that preserves semantic roles while maintaining syntactic fidelity, supported by extensive statistics and visualizations. The dataset and tooling are openly released, enabling reproducible cross-language representation learning and advanced program analysis.

Abstract

We introduce the MultiLang Code Parser Dataset (MLCPD), a large-scale, language-agnostic dataset unifying syntactic and structural representations of code across ten major programming languages. MLCPD contains over seven million parsed source files normalized under our proposed universal Abstract Syntax Tree (AST) schema, enabling consistent cross-language reasoning, structural learning, and multilingual software analysis. Unlike existing corpora that focus purely on token-level code or isolated parsers, MLCPD provides both hierarchical tree representations and rich metadata for every file, ensuring lossless syntactic coverage and structural uniformity. Each entry includes a normalized schema, language-level metadata, and abstracted node semantics stored in Parquet format for scalable retrieval. Empirical analyses reveal strong cross-language structural regularities-demonstrating that syntactic graphs from languages as diverse as Python, Java, and Go can be aligned under a shared schema. We release the dataset publicly on Hugging Face and the accompanying codebase on GitHub, which includes complete pipelines for dataset reproduction, grammar compilation, and a visualization tool for exploring the unified AST across languages. Together, these resources establish MLCPD as an open, reproducible foundation for future research in cross-language representation learning and program analysis.

MLCPD: A Unified Multi-Language Code Parsing Dataset with Universal AST Schema

TL;DR

MLCPD addresses cross-language code understanding by providing a unified, lossless AST-based representation across ten languages. It constructs over seven million parsed files with a four-layer universal schema and Tree-sitter-driven parsing, enabling scalable, language-agnostic analyses. The authors demonstrate cross-language structural regularities and robust mapping that preserves semantic roles while maintaining syntactic fidelity, supported by extensive statistics and visualizations. The dataset and tooling are openly released, enabling reproducible cross-language representation learning and advanced program analysis.

Abstract

We introduce the MultiLang Code Parser Dataset (MLCPD), a large-scale, language-agnostic dataset unifying syntactic and structural representations of code across ten major programming languages. MLCPD contains over seven million parsed source files normalized under our proposed universal Abstract Syntax Tree (AST) schema, enabling consistent cross-language reasoning, structural learning, and multilingual software analysis. Unlike existing corpora that focus purely on token-level code or isolated parsers, MLCPD provides both hierarchical tree representations and rich metadata for every file, ensuring lossless syntactic coverage and structural uniformity. Each entry includes a normalized schema, language-level metadata, and abstracted node semantics stored in Parquet format for scalable retrieval. Empirical analyses reveal strong cross-language structural regularities-demonstrating that syntactic graphs from languages as diverse as Python, Java, and Go can be aligned under a shared schema. We release the dataset publicly on Hugging Face and the accompanying codebase on GitHub, which includes complete pipelines for dataset reproduction, grammar compilation, and a visualization tool for exploring the unified AST across languages. Together, these resources establish MLCPD as an open, reproducible foundation for future research in cross-language representation learning and program analysis.
Paper Structure (16 sections, 7 figures, 4 tables, 2 algorithms)

This paper contains 16 sections, 7 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Design and Implementation
  4. Data Collection and Preprocessing
  5. Design Philosophy and Universal Schema
  6. Layer 1: Metadata Block.
  7. Layer 2: Flat Node Array.
  8. Layer 3: Node Categorization.
  9. Layer 4: Cross-Language Map.
  10. Empirical Schema Validation and Parser Evolution
  11. Parsing and Normalization Pipeline
  12. Dataset Statistics and Structural Insights
  13. Cross-Language Structural Correlation
  14. Qualitative Example: Python vs. Java “Age Check” Schema
  15. Discussion and Limitations
  16. ...and 1 more sections

Figures (7)

  • Figure 1: Visualization of Python and Java "Age Check" programs parsed under the universal schema, demonstrating consistent alignment across languages.
  • Figure 2: Distribution of records across ten programming languages. The uniform proportions validate balanced representation.
  • Figure 3: Average AST node count per file across languages. Higher counts reflect verbose syntactic forms and explicit typing.
  • Figure 4: Average node density per language, reflecting syntactic compactness and complexity.
  • Figure 5: Disk versus memory footprint per language, confirming consistent compression ratios across corpora.
  • ...and 2 more figures