Parsing TTree Formula in Python

TL;DR

The paper addresses the lack of a formal, efficient parser for ROOT's TTreeFormula within Python tools like Uproot. It introduces Formulate, a Lark-based, zero-dependency parser that defines TTreeFormula and NumExpr grammars in EBNF, builds a language-agnostic AST, and supports emission to multiple targets (TTreeFormula, NumExpr, Python). The work delivers a linear-time, pure-Python solution with a formal grammar for TTreeFormula, enabling large-scale expressions to be parsed up to $526$-fold faster on real-world inputs, and integrates seamlessly with Uproot to resolve aliases directly from ROOT files. This enhances interoperability between ROOT data structures and Python-based data analysis, with practical impact on performance and usability.

Abstract

Uproot can read ROOT files directly in pure Python but cannot (yet) compute expressions in ROOT's TTreeFormula expression language. Despite its popularity, this language has only one implementation and no formal specification. In a package called "formulate," we defined the language's syntax in standard BNF and parse it with Lark, a fast and modern parsing toolkit in Python. With formulate, users can now convert ROOT TTreeFormula expressions into NumExpr and Awkward Array manipulations. In this contribution, we describe BNF notation and the Look Ahead Left to Right (LALR) parsing algorithm, which scales linearly with expression length. We also present the challenges with interpreting TTreeFormula expressions as a functional language; some function-like forms can't be expressed as true functions. We also describe the design of the abstract syntax tree that facilitates conversion between the three languages. The formulate package has zero package dependencies, so we are adding it as one of Uproot's dependencies so that Uproot will be able to use TTreeFormula expressions, whether they are hand-written or embedded in a ROOT file as TTree aliases.

Figures (3)

• Figure 1: The flow for transpiling the three languages is illustrated in this diagram. The library can parse either TTreeFormula or NumExpr using the from_root or the from_numexpr methods. Once parsed, an expression tree is generated by the Lark parser, which is converted into an Abstract Syntax Tree by calling the to_AST method internally. Once we have the AST and it's nested nodes, the user can call either the to_root, to_numexpr or to_python to generate strings of the translated expression.
• Figure 2: The EBNF for both TTreeFormula and NumExpr. The parts in Grey denote the grammar for parsing NumExpr while the whole text defines the grammar for TTreeFormula. Please note that the NumExpr grammar defines func_name as CNAME, after which the NumExpr grammar terminates.
• Figure 3: The previous implementation of Formulate scales exponentially with the number of tokens in the parsing expression whereas the curve for the new implementation is much less aggressive. The y-axis is on a log scale. The x-axis indicates how many times the original expression (54 characters) has been repeated