Intelligent Spark Agents: A Modular LangGraph Framework for Scalable, Visualized, and Enhanced Big Data Machine Learning Workflows

TL;DR

This work presents a Spark-based modular LangGraph framework that uses Agent AI to automate data preprocessing, feature engineering, and model evaluation within graph-structured workflows. By integrating Spark with LangChain and LangGraph, the approach enables visual design of ML pipelines and automatic translation into high-performance Spark code, augmented by Spark SQL/DataFrame agents and LLM-assisted reasoning. Key contributions include a frame-based intermediate data storage design with Alluxio, SparkMLlib-driven componentization of ML steps, rigorous DAG creation and validation, and translation techniques for complex parallel pipelines. Experimental results on real datasets demonstrate feasibility, improved scalability, and potential accuracy gains, underscoring the practical impact of scalable, intelligent, graph-guided data analytics in big data environments.

Abstract

This paper presents a Spark-based modular LangGraph framework, designed to enhance machine learning workflows through scalability, visualization, and intelligent process optimization. At its core, the framework introduces Agent AI, a pivotal innovation that leverages Spark's distributed computing capabilities and integrates with LangGraph for workflow orchestration. Agent AI facilitates the automation of data preprocessing, feature engineering, and model evaluation while dynamically interacting with data through Spark SQL and DataFrame agents. Through LangGraph's graph-structured workflows, the agents execute complex tasks, adapt to new inputs, and provide real-time feedback, ensuring seamless decision-making and execution in distributed environments. This system simplifies machine learning processes by allowing users to visually design workflows, which are then converted into Spark-compatible code for high-performance execution. The framework also incorporates large language models through the LangChain ecosystem, enhancing interaction with unstructured data and enabling advanced data analysis. Experimental evaluations demonstrate significant improvements in process efficiency and scalability, as well as accurate data-driven decision-making in diverse application scenarios. This paper emphasizes the integration of Spark with intelligent agents and graph-based workflows to redefine the development and execution of machine learning tasks in big data environments, paving the way for scalable and user-friendly AI solutions.

Figures (12)

• Figure 1: Typical machine learning process.
• Figure 2: System architecture diagram and flow chart.
• Figure 3: Architectural Design of Spark Agent Based on LangGraph.
• Figure 4: Component class design and inheritance diagram.
• Figure 5: Translation Method for Multiple Join/fork Parallel Tasks.