Bridging Worlds: Achieving Language Interoperability between Julia and Python in Scientific Computing
Ianna Osborne, Jim Pivarski, Jerry Ling
TL;DR
Bridging Worlds investigates language interoperability between Julia and Python within High Energy Physics data analysis, focusing on Awkward Array as a cross-language bridge. The work analyzes memory management, data buffer sharing, and dependency coordination, demonstrating performance gains by invoking Julia from Python and vice versa for array-oriented workloads. Key contributions include memory-sharing via AwkwardArray.jl, custom data-conversion rules in PythonCall, and concrete CMS data-analysis workflows illustrating end-to-end cross-language pipelines. Overall, the paper shows how combining Julia's performance with Python's ecosystem enables scalable, cross-language scientific computing in HEP contexts.
Abstract
In the realm of scientific computing, both Julia and Python have established themselves as powerful tools. Within the context of High Energy Physics (HEP) data analysis, Python has been traditionally favored, yet there exists a compelling case for migrating legacy software to Julia. This article focuses on language interoperability, specifically exploring how Awkward Array data structures can bridge the gap between Julia and Python. The talk offers insights into key considerations such as memory management, data buffer copies, and dependency handling. It delves into the performance enhancements achieved by invoking Julia from Python and vice versa, particularly for intensive array-oriented calculations involving large-scale, though not excessively dimensional, arrays of HEP data. The advantages and challenges inherent in achieving interoperability between Julia and Python in the domain of scientific computing are discussed.