What If We Had Used a Different App? Reliable Counterfactual KPI Analysis in Wireless Systems

TL;DR

The paper tackles reliable counterfactual KPI analysis in wireless systems, where a non-real-time controller logs context, app, and KPI data $(x,a,y)$ and seeks KPIs $y_{a'}$ that would have occurred under a different app $a'$. It introduces Counterfactual Conformal KPI Estimation (CCKE), a post-hoc framework based on Weighted Conformal Prediction to produce finite-sample-valid prediction sets for multiple KPIs under covariate shift via density-ratio weighting. The method splits data by target app, trains a quantile regression model, and calibrates intervals using calibration data to guarantee coverage; theoretical guarantees hold with exact weights, and degrade gracefully with weight estimation error. Empirically, CCKE is validated on two wireless scenarios—MAC-layer scheduling and physical-layer MIMO transmission—demonstrating reliable counterfactual intervals where baselines fail to maintain the desired coverage, and showing scalable computation. This work provides practical, reliable tools for diagnosing, explaining, and optimizing app-selection strategies in O-RAN-like architectures, enabling data-driven What-If analyses without environment interventions.

Abstract

In modern wireless network architectures, such as Open Radio Access Network (O-RAN), the operation of the radio access network (RAN) is managed by applications, or apps for short, deployed at intelligent controllers. These apps are selected from a given catalog based on current contextual information. For instance, a scheduling app may be selected on the basis of current traffic and network conditions. Once an app is chosen and run, it is no longer possible to directly test the key performance indicators (KPIs) that would have been obtained with another app. In other words, we can never simultaneously observe both the actual KPI, obtained by the selected app, and the counterfactual KPI, which would have been attained with another app, for the same network condition, making individual-level counterfactual KPIs analysis particularly challenging. This what-if analysis, however, would be valuable to monitor and optimize the network operation, e.g., to identify suboptimal app selection strategies. This paper addresses the problem of estimating the values of KPIs that would have been obtained if a different app had been implemented by the RAN. To this end, we propose a conformal-prediction-based counterfactual analysis method for wireless systems that provides reliable error bars for the estimated KPIs, despite the inherent covariate shift between logged and test data. Experimental results for medium access control-layer apps and for physical-layer apps demonstrate the merits of the proposed method.

Figures (9)

• Figure 1: In the wireless system under study, the radio access network (RAN) is managed by a non-real-time controller. The controller collects data from the RAN about key performance indicators (KPIs) attained by apps implemented by the RAN. Accordingly, the controller logs data in the form $(x,a,y)$ as the data set, where $x$ is the context, $a$ is the app identifier, and $y$ is the KPIs. The controller implements the counterfactual analysis by answering a what-if question: Given that app $a$ has obtained KPIs $y$ for context $x$, what would the KPIs have been for the same context $x$ had some other app $a'\neq a$ been selected by the non-real-time controller?
• Figure 2: (a) Uplink resource allocation problem in which, based on the initial backlogs and channel quality indicators (CQIs), the non-real-time controller chooses a scheduling app. (b) Given initial backlogs $b^{\mathrm{in}} = [b^{\mathrm{in}}_1,\dots,b^{\mathrm{in}}_K]$ and CQIs $c = [c_1,\dots,c_K]$ for the $K$ users, assume that the controller has selected the proportional fair channel aware (PFCA) scheduling app, which has produced a final backlog $b^{\mathrm{fin}} = [b^{\mathrm{fin}}_1,\dots,b^{\mathrm{fin}}_K]$. What would the backlog have been if a round-robin (RR) scheduling had been selected instead?
• Figure 3: In a multi-antenna communication link, based on the context $x$ encompassing the initial average SNR and information about the propagation environment, the non-real-time controller chooses a transmission app $a$. The transmission app $a$ is determined by a multiplexing-based or diversity-based space-time method, along with a constellation. Assuming that the controller has selected a multiplexing-based scheme with a QPSK constellation, what would the latency have been if the transmitter had used a diversity-based scheme with QPSK?
• Figure 4: Overview of CCKE: (a) The RAN collects a data set $\mathcal{D}$ consisting of context $x_n$, selected app $a_n$, and corresponding observed KPIs $y_n$ for $n=1,...,N$. (b) Given the new context $x$, the controller selects app $a$ and the RAN observes the corresponding KPIs $y$. CCKE aims at reliably estimating the observed KPIs if app $a'\neq a$ had been chosen instead of $a$ for the current context $x$. (c) To do so, CCKE first extracts data set $\mathcal{D}_{a'}$ from the entire data set $\mathcal{D}$ by choosing only the data points corresponding to app $a'$. Then, the first split of the data set is used to train the predictor (\ref{['CQR']}), while the remaining data are used for calibration via (\ref{['NC score']}). The resulting calibrated prediction interval (\ref{['w_interval_calculate']}) provably contains the true KPIs $y_{a'}$ for the target app $a'$ of interest (Lemma \ref{['lemma']}).
• Figure 5: Illustration of the app selection probability (\ref{['e_x']}) applied by the non-real-time controller for the medium access control scheduling example.

Theorems & Definitions (3)

• Lemma 1: Coverage guarantee of CCKE
• Lemma 2: Coverage guarantee of CCKE with estimated weight $\hat{w}_{a'\rightarrow a}(x)$
• Lemma 3: Coverage guarantee of CCKE with noisy KPI measurements