Evaluating Learned Query Performance Prediction Models at LinkedIn: Challenges, Opportunities, and Findings

TL;DR

This paper tackles the challenge of predicting industrial query performance on LinkedIn's Trino-based OLAP workloads with limited execution plan information. It systematically compares TLSTM, TCNN, QueryFormer, and XGBoost on latency regression, CPU-time prediction, and a classification task, introducing a multi-task learning extension to exploit operator-level metrics. Key findings show that QueryFormer with multi-task learning excels in regression, while XGBoost dominates classification and offers the fastest inference, with CPU-time predictions consistently more accurate than latency. The work also derives an empirical upper bound on achievable accuracy due to lossy plan encoding, revealing that gains saturate even with more data, and highlighting the practical implications for model selection and feature utilization in industrial settings. $P_{50}$, $P_{90}$, and $P_{99}$ quantile q-errors are used to quantify predictive performance and bound considerations.$

Abstract

Recent advancements in learning-based query performance prediction models have demonstrated remarkable efficacy. However, these models are predominantly validated using synthetic datasets focused on cardinality or latency estimations. This paper explores the application of these models to LinkedIn's complex real-world OLAP queries executed on Trino, addressing four primary research questions: (1) How do these models perform on real-world industrial data with limited information? (2) Can these models generalize to new tasks, such as CPU time prediction and classification? (3) What additional information available from the query plan could be utilized by these models to enhance their performance? (4) What are the theoretical performance limits of these models given the available data? To address these questions, we evaluate several models-including TLSTM, TCNN, QueryFormer, and XGBoost, against the industrial query workload at LinkedIn, and extend our analysis to CPU time regression and classification tasks. We also propose a multi-task learning approach to incorporate underutilized operator-level metrics that could enhance model understanding. Additionally, we empirically analyze the inherent upper bound that can be achieved from the models.

Figures (12)

• Figure 1: Trino queries latency distribution with log scale
• Figure 2: Deep learning models operator-level encoding example
• Figure 3: Deep learning models plan-level encoding example
• Figure 4: XGBoost encoding example
• Figure 5: Multi-task learning example