An Innovative Algorithm For Robust, Interactive, Piecewise-Linear Data Exploration
Stephen Wright, Colin Paterson
TL;DR
This work addresses the challenge of robust, explicable parameter estimation for short, noisy, multidimensional economic data with potential regime shifts. It introduces a mode-based generalization of the Theil-Sen-Siegel algorithm to produce $piecewise-linear$ fits, augmented with $LAD ext{-}LASSO$ regularization, cross-validation, and a non-parametric $Hamming ext{-}Distance$ affinity-cluster framework to detect multiple data-generating processes. Key contributions include axis-angle mode projections, Siegel-style mode partitioning, a cluster-affinity approach that yields block-diagonal structure, and an interactive browser-based demonstration. The approach aims to unify robust regression, clustering, and regularisation in a single, distribution-free pipeline suitable for explicable analysis of small datasets, with potential applicability to real-world economic datasets and beyond.
Abstract
Many mathematical modelling tasks (such as in Economics and Finance) are informed by data that is "found" rather than being the result of carefully designed experiments. This often results in data series that are short, noisy, multidimensional and contaminated with outliers, regime shifts, and confounding, uninformative or co-linear variables. We present a generalization of the Theil-Sen algorithm to reflect modes (rather than the median) in the parameter space distribution (of partial fits to the data). This can provide a robust piecewise-linear fit to the data while also allowing for extensions to including elements of cluster analysis, regularization and cross-validation in a unified (distribution free) approach that can:- 1. Exploit piecewise linearity to reduce the need to pre-specify the form of the underlying data generating process. 2. Detect non-homogeneity (e.g. regime shifts, multiple data generating processes etc.) in the data using an innovative non-parametric (Hamming-Distance/Affinity-Matrix) cluster analysis technique. 3. Enable dimension reduction and resistance to the effects of multi-co-linearity by including LASSO regularization as an integral part of the algorithm. 4. Estimate measures of accuracy, such as standard errors, bias, and confidence intervals, without needing to rely on traditional distributional assumptions. Taken together these extensions to the traditional Theil-Sen algorithm simplify the traditional process of parameter fitting by providing a single-stage analysis controlled by a multidimensional search of Scale/Parsimony/Precision hyper-parameters. These are early days in this research and the main limitation in this approach is that it assumes that compute power is infinite and compute time is small enough to allow interactive use.