Inexact Simplification of Symbolic Regression Expressions with Locality-sensitive Hashing

TL;DR

This paper tackles the bloat problem in symbolic regression by introducing an inexact, data-driven simplification method based on locality-sensitive hashing (LSH). Subtrees are memoized in a table keyed by SimHash of their prediction vectors, and replacements are made with smaller equivalent subtrees when the semantic distance falls within a threshold $0.01$, enabling continual simplification during evolution. Empirical results show improved convergence and lower complexity with comparable or better accuracy across several datasets, at the cost of additional optimization time per generation. The method learns both general and problem-specific simplification rules without handcrafted algebraic identities, offering a scalable, agnostic approach to bloat control in SR with potential for integration into existing genetic programming frameworks.

Abstract

Symbolic regression (SR) searches for parametric models that accurately fit a dataset, prioritizing simplicity and interpretability. Despite this secondary objective, studies point out that the models are often overly complex due to redundant operations, introns, and bloat that arise during the iterative process, and can hinder the search with repeated exploration of bloated segments. Applying a fast heuristic algebraic simplification may not fully simplify the expression and exact methods can be infeasible depending on size or complexity of the expressions. We propose a novel agnostic simplification and bloat control for SR employing an efficient memoization with locality-sensitive hashing (LHS). The idea is that expressions and their sub-expressions traversed during the iterative simplification process are stored in a dictionary using LHS, enabling efficient retrieval of similar structures. We iterate through the expression, replacing subtrees with others of same hash if they result in a smaller expression. Empirical results shows that applying this simplification during evolution performs equal or better than without simplification in minimization of error, significantly reducing the number of nonlinear functions. This technique can learn simplification rules that work in general or for a specific problem, and improves convergence while reducing model complexity.

Figures (10)

• Figure 1: Given a set of vectors (i.e. the predictions), similar predictions are mapped to the same hash. The two highlighted vectors present close values in this example and are mapped into the same hash.
• Figure 2: In the first stage ($1$), we have a simplification table with only the problem variables plus the constant. Every node is seen as the root of a subtree and can generate a prediction vector. The second stage ($2$) uses the predictions to get hash values for each node, updating the simplification table. Finally, we get the simplified tree by replacing the nodes with the smallest subtree of the same hash in the simplification table.
• Figure 3: Validation error of the best individual during the evolution. $y$ axis is in log-scale.
• Figure 4: Number of simplifications performed in each generation.
• Figure 5: Final size of the solutions found by each method. The simplification methods showed no statistically different significance related to without any simplification.