Transfer Q-learning

TL;DR

This work develops Transfer Q-learning for time-inhomogeneous finite-horizon MDPs to address high-dimensional states and limited data. By introducing re-targeting of source pseudo-responses and a backward-inductive estimation, the method enables both cross-task and cross-stage transfer, with theoretical guarantees for faster offline convergence and reduced offline-to-online regret under reward-function similarity. The framework supports offline-to-offline and offline-to-online transfer, as well as streaming retargeting, and provides a linear-function-approximation instantiation with convergence rates that improve over single-task Q-learning when source data are informative. These results have practical implications for data-scarce, multi-stage RL domains such as healthcare and business, where leveraging related tasks can substantially accelerate learning and improve decision quality.

Abstract

Time-inhomogeneous finite-horizon Markov decision processes (MDP) are frequently employed to model decision-making in dynamic treatment regimes and other statistical reinforcement learning (RL) scenarios. These fields, especially healthcare and business, often face challenges such as high-dimensional state spaces and time-inhomogeneity of the MDP process, compounded by insufficient sample availability which complicates informed decision-making. To overcome these challenges, we investigate knowledge transfer within time-inhomogeneous finite-horizon MDP by leveraging data from both a target RL task and several related source tasks. We have developed transfer learning (TL) algorithms that are adaptable for both batch and online $Q$-learning, integrating valuable insights from offline source studies. The proposed transfer $Q$-learning algorithm contains a novel {\em re-targeting} step that enables {\em cross-stage transfer} along multiple stages in an RL task, besides the usual {\em cross-task transfer} for supervised learning. We establish the first theoretical justifications of TL in RL tasks by showing a faster rate of convergence of the $Q^*$-function estimation in the offline RL transfer, and a lower regret bound in the offline-to-online RL transfer under stage-wise reward similarity and mild design similarity across tasks. Empirical evidence from both synthetic and real datasets is presented to evaluate the proposed algorithm and support our theoretical results.

Figures (1)

• Figure 1: Illustration of single-task $Q$-learning and transfer $Q$-learning. Naive application of transfer algorithms for regression without re-targeting will incur extra accumulated bias along the red arrows.

Theorems & Definitions (4)

• Remark 1
• Theorem 4.2: Convergence rate of Algorithm \ref{['alg-linear']}
• Remark 2: Convergence rate of single-task $Q$-learning
• Remark 3: Similarity characterization with $q\in[0,1)$] In Theorem \ref{['thm1-tl']}, we require $\boldsymbol{\delta}_t^{(k)}$ to be approximately sparse ($q=1$). For $q\in[0,1)$, transferred Q-learning algorithms can be analogously developed based on Algorithm 1 in the supplement of li2022transfer-jrssb, which is a minimax optimal approach for transfer learning in linear models for $q\in[0,1)$. Our theoretical guarantees rely on stage-wise reward similarity together with mild design similarity across tasks. If transition dynamics differ substantially, thereby violating the covariance-similarity condition, or if the reward discrepancies $h$ are large, the advantages of transfer learning may diminish or even disappear. These extensions call for new estimation tools and theoretical analysis, but they provide a principled roadmap for expanding our framework beyond the reward-similarity regime studied here. A full development of these methods is beyond the scope of this paper and will be pursued in future work. In this subsection, we provide theoretical guarantees for the online Algorithm \ref{['alg-online']} with knowledge transferred from offline data for $M=2$. Results for a finite $M$ can be derived similarly, with the final outcomes differing only by a constant factor dependent on $M$. In the online setting, the learner aims to minimize the cumulative regret that measures the expected loss of following the estimated optimal policy instead of the oracle optimal policy. Mathematically, the cumulative regret over $T$ episodes is defined as ${\rm Regret}_{NT} = \sum_{i=1}^N \sum_{t=1}^T \gamma^t \left( \mathbb{E}[r_{t,i}^{(0)}|\boldsymbol{s}^{(0)}_{t,i},a^*_{t,i}]-\mathbb{E}[r_{t,i}^{(0)}|\boldsymbol{s}^{(0)}_{t,i},\widehat{a}^{(0)}_{t,i}] \right),$ where the estimated optimal policy is $\widehat{a}_{t,i}^{(0)} = \mathop{\mathrm{arg\,max}}\limits_{a'\in\{-1,1\}}Q(\boldsymbol{s}_{t,i}^{(0)},a';\widehat{\boldsymbol{\theta}}^{(0)}_t)$ and the oracle optimal policy is $a^*_{t,i} = \mathop{\mathrm{arg\,max}}\limits_{a'\in\{-1,1\}}Q(\boldsymbol{s}_{t,i}^{(0)},a'; \boldsymbol{\theta}^{(0)}_t)$. Under the linear $Q^*$-function \ref{['eqn:q-k-linear']} with $M=2$, they can be further simplified to $\widehat{a}_{t,i}^{(0)} = sgn((\boldsymbol{s}^{(0)}_{t,i})^{\top}\widehat{\boldsymbol{\psi}}_t), \quad\text{and}\quad a^*_{t,i}=sgn((\boldsymbol{s}^{(0)}_{t,i})^{\top}\boldsymbol{\psi}_t),$ where $sgn(\cdot)$ is the sign function. The regret bound of online learning with offline transfer is given in Theorem \ref{['thm-ol']}. Suppose that Assumptions \ref{['assume-subgaussian-eigenval']} hold, (\ref{['eq-delta-sparse']}) holds with $q=1$. and $s\sqrt{\log p/N_{\mathrm{src}}}+h\leq C$, $n_e\gtrsim s^2h^2\log p+\log p$. We take tuning parameters to be $\lambda_{\mathrm{src}} =\sqrt{\frac{\log p}{N_{\mathrm{src}}}} + \sqrt{\frac{h}{s}} \left( \frac{\log p}{n_e} \right)^{1/4} ~~\text{and}~~ \lambda_0=\sqrt{\frac{\log p}{n_e}}.$ For any $N>n_e$, {\rm Regret}_{NT} \lesssim \frac{n_e\gamma }{1-\gamma} + (N-n_e) \left( \sqrt{\frac{s\log p}{N_{\mathrm{src}}}}+ h^{1/2} \left( \frac{\log p}{n_e} \right)^{1/4} \right) with probability at least $1-\exp(-c_1\log p)$. We now find the optimal choice of $n_e$, which minimizes the RHS of (\ref{['reg-tl']}). To simplify the analysis, we parameterize $h=N_{\mathrm{src}}^{-\alpha}$ for some $\alpha\geq 0$. In the supplementary files, we show that if we take n_e \asymp \max\left\{\frac{N^{4/5}(\log p)^{1/5}}{N_{\mathrm{src}}^{2\alpha/5}},\frac{s^2\log p}{N_{\mathrm{src}}^{2\alpha}}\right\}, then with probability at least $1-\exp(-c_1\log p)$ ${\rm Regret}_{NT} \lesssim \max\left\{\frac{N^{4/5}(\log p)^{1/5}}{N_{\mathrm{src}}^{2\alpha/5}},\frac{s^2\log p}{N_{\mathrm{src}}^{2\alpha}}\right\}+N\sqrt{\frac{s\log p}{N_{\mathrm{src}}}}.$ We see that the larger the $N_{\mathrm{src}}$, i.e., more source data, the smaller the cumulative regret; and the larger the value of $\alpha$, i.e., higher the similarity, the smaller the cumulative regret. Without offline data, we denote the estimated optimal policy by $\widehat{a}^{(st)}$. It is easy to show that for $n_e\gg (s\log p)^2$, then with probability at least $1-\exp(-c_1\log p)$, {\rm Regret}_{NT}^{(st)} \lesssim \frac{n_e\gamma}{1-\gamma}+(N-n_e)\sqrt{\frac{s\log p}{n_e}}. with probability at least $1-\exp(-c_1\log p)$. If we take $n_e\asymp N^{2/3}(s\log p)^{1/3}\vee (s\log p)^2$, then {\rm Regret}_{NT}^{(st)}\lesssim N^{2/3}(s\log p)^{1/3}+ (s\log p)^2, with probability at least $1-\exp(-c_1\log p)$. Comparing \ref{['reg-tl']} with \ref{['reg-st']}, we see that the cumulative regret of transfer $Q$-learning is always smaller when $h=o(s\sqrt{\log p/n_e})$. Comparing \ref{['reg-tl-opt']} with \ref{['reg-st-opt']}, we see that as long as $N_{\mathrm{src}}>N^{1/(3\alpha)}+N^{2/3}$, the regret of transfer learning policy is always no larger than the single-task policy. This comparison further implies that if $N$ is very large, then the optimal choice of $n_e$ can be larger than $N_{\mathrm{src}}$. In this case, transfer learning may not yield a significant improvement and it suffices to consider single-task $Q$-learning. In this section, we demonstrate the advantages of the proposed transferred Q-learning algorithm on synthetic and real data sets. Code and implementation details for this section are available online. We first consider a simple MDP model which has an analytical form for the optimal $Q^*$ function. In such a setting we can explicitly compare the estimated $Q^*$ function with the ground truth. The generative model is designed based on those in chakraborty2010inference and song2015penalized. The underlying model is a two-stage MDP with two possible actions ${\cal A}=\left\{ -1,1 \right\}$ and two states ${\cal S} = \left\{ -1,1 \right\}$. The binary states $S_t$ and the binary treatment $A_t$ are generated as follows: $\Pr\left( S_1=-1 \right) = \Pr\left( S_0=1 \right) = 0.5,\Pr\left( A_t=-1 \right) = \Pr\left( A_t=1 \right) = 0.5, \quad t = 1, 2,\Pr\left( S_2 | S_1, A_1 \right) = 1 - \Pr\left( S_2 | S_1, A_1 \right) = {\rm expit}\left( b_1 S_1 + b_2 A_1 \right),$ where ${\rm expit}\left( x \right) = \exp\left( x \right) / \left( 1 + \exp\left( x \right) \right)$. The immediate reward $R_1 = 0$ and $R_2$ is given by $R_2 = \kappa_1 + \kappa_2 S_1 + \kappa_3 A_1 + \kappa_4 S_1 A_1 + \kappa_5 A_2 + \kappa_6 S_2 A_2 + \kappa_7 A_1 A_2 + \varepsilon_2,$ where $\varepsilon_2 \sim {\cal N}\left( 0, 1 \right)$. Under this setting, the true $Q^*$ functions for time $t=1, 2$ are $Q_2\left( S_2, A_2; \boldsymbol{\theta}_2 \right)= \theta_{2,1} + \theta_{2,2} S_1 + \theta_{2,3} A_1 + \theta_{2,4} S_1 A_1~~~ + \theta_{2,5} A_2 + \theta_{2,6} S_2 A_2 + \theta_{2,7} A_1 A_2Q_1\left( S_1, A_1; \boldsymbol{\theta}_1 \right)= \theta_{1,1} + \theta_{1,2} S_1 + \theta_{1,3} A_1 + \theta_{1,4} S_1 A_1,$ where the true coefficients $\boldsymbol{\theta}_t$ are explicitly functions of $b_1$, $b_2$, $\kappa_1,\dots, \kappa_7$ given in \ref{['eqn:true-q-theta']} in the supplemental material. At each state, the observed covariates $X_t\in \mathbb{R}^p$, $p=100$, consist of first eight elements $1$, $S_1$, $A_1$, $S_1A_1$, $A_2$, $S_2$, $S_2 A_2$, $A_1 A_2$ and the remaining elements that are randomly sampled from standard normal. We consider transfer between two different but similar MDPs from the above model. The target and source MDPs are different in the coefficients $\kappa$'s and therefore $\theta$'s in \ref{['eqn:true-q']}. Specifically, we set $\theta_{2,j} = 1$, $1 \le j \le 7$ for the target MDP. The second-stage coefficients of the source task are the same except that the second element $\theta_{2,2}^{(1)} = 1.2$. Therefore, according to equation \ref{['eqn:true-q-theta']} in the supplemental material, the true coefficients of stage-one $Q^*$ functions are $\theta_{1,1}, \theta_{1,2}, \theta_{1,3}, \theta_{1,4} \approx 2.69, 1.19, 1.69, 1.19$ for the target MDP, and $\theta_{1,1}^{(1)}, \theta_{1,2}^{(1)}, \theta_{1,3}^{(1)}, \theta_{1,4}^{(1)} \approx 2.69, 1.39, 1.69, 1.19$ for the source MDP. In summary, the true $Q_1$ functions of the target and source MDPs are different only in $\theta_{1,2}$ and the $Q_2$ functions are different only in $\theta_{2,2}$. The coefficients of the remaining elements in $X_t$ are all set to zero. We generate trajectories of the form $\left( \boldsymbol{x}_{1,i}, a_{1,i}, r_{1,i}, \boldsymbol{x}_{2,i}, a_{2,i}, r_{2,i} \right)$ from both target and source MDPs. The target task consists of $n_0$ trajectories while the source task consists of $n_1$ trajectories. We compare $\left\lVert\widehat{\boldsymbol{\theta}}_t - \boldsymbol{\theta}_t\right\rVert_2^2$, $t=1,2$, with or without transfer under combinations of $n_0$ and $n_{\mathrm{src}}=n_1$. The boxplots are shown in Figure \ref{['fig:chakra-coef']}. We also generate a new dataset and compare predicted $\widehat{Q}^*(\boldsymbol{x}, a)$ obtained by transferred $Q^*$ learning and its vanilla counterpart. The boxplots of $\left\lvert\widehat{Q}^*(\boldsymbol{x}, a) - Q^*(\boldsymbol{x}, a)\right\rvert/\left\lvert Q^*(\boldsymbol{x}, a)\right\rvert$ (averaged over all state-action pairs in the new dataset) are presented in Figure \ref{['fig:chakra-pred']}. It is clear that transfer $Q$-learning performs much better than the vanilla $Q$-learning without transfer in terms of both coefficient estimation and prediction. The advantage is more prominent in earlier stages since the transfer benefit in the latter stage positively cascade to the earlier stages through the second term in \ref{['eq-mt']}. Table \ref{['tab:opt-act']} compares the frequency of correct optimal actions chosen by single-task $Q$-learning and transfer $Q$-learning with the new dataeset. We observe that transfer $Q$-learning achieves higher accuracy in choosing the optimal actions across all combinations of $n_0$ and $n_1$, which are the number of trajectories of the target task and the source task, respectively. The amplitude of accuracy increase is higher when $n_0$ is small. This again shows the advantage of transfer $Q$-learning, especially when the number of trajectories of the target task is small. Comparison of the estimated coefficients of the optimal $Q^*$ function. The $y$-axis represents $\left\lVert\widehat{\boldsymbol{\theta}}_t - \boldsymbol{\theta}_t\right\rVert_2^2$ for $t=1$ (left) and $t=2$ (right). The dimension is $p =100$ and sparsity $s = 7$. Comparison of the prediction of the optimal $Q^*$ function with different sample size configurations. The $y$-axis represents $\left\lvert\widehat{Q}^* - Q^*\right\rvert/\left\lvert Q^*\right\rvert$. The dimension is $p =100$ and sparsity $s = 7$. The frequency of correct optimal actions. The comparison is between the optimal action $A^{*,(st)}$ chosen by single-task $Q$-learning and the optimal action $A^{*,(tl)}$ chosen by transfer $Q$-learning. We consider different combinations of $n_0 = 30, 50, 70$ and $n_1 = n_0 + 10, \, \cdots, n_0 + 50$, where $n_0$ and $n_1$ denote the number of trajectories of the target task and the source task, respectively. $n_0$$A^{*,(st)}$$A^{*,(tl)}$ with different $n_1$$n_0 + 10$$n_0 + 20$$n_0 + 30$$n_0 + 40$$n_0 + 50$300.5550.9650.9850.9900.9700.985500.8600.9900.9951.0001.0001.000700.90511111 In this section, we study the empirical performance of online transfer $Q$-learning using an offline source dataset. The generative MDP is the same as that defined in \ref{['eqn:true-q']}. We have access to an offline source dataset of trajectories $\left( X_1, A_1, R_1, X_2, A_2, R_2 \right)$ generated from an MDP with $\theta_{2j}^{(1)} = \kappa_j^{(1)} = 1$, $1 \le j \le 3$, $5 \le j \le 7$ and $\theta_{2j}^{(1)} = \kappa_j^{(1)} = 2$, $j=4$. The online target RL task is modeled by an MDP with $\theta_{2j}^{(1)} = \kappa_j^{(1)} = 1$, $1 \le j \le 7$. The dimension of covariates is $p = 100$. We first study the cumulative regret of transferred and vanilla $Q$-learning with different lengths of exploration, $n_{e} \in \left\{ 1, 2, \cdots, 20 \right\}$. The size of the offline source dataset is $n_1= 100$. At the exploration stage, $n_{e}$ trajectories are generated by random actions. The coefficients $\widehat{\boldsymbol{\theta}}_t$ are estimated using $Q$-learning with or without transfer. The average cumulative regret at the exploitation stage versus different length of exploration are presented in Figure \ref{['fig:chakra-online']} (a). The length of exploitation stage is $100$. For each $n_{e}$, the mean of the cumulative regret at exploitation stage is reported since the values of $\widehat{\boldsymbol{\theta}}_t^{(0)}$ are not updated during exploitation stage. We observe that under the condition that $h\ll s\sqrt{\log p/n_{e}}$, the regret of transfer $Q$-learning is much smaller than that of vanilla $Q$-learning, which is consistent with the result in Theorem \ref{['thm-ol']}. Since Algorithm \ref{['alg-online']} is of the explore-then-commit (ETC) type, the advantage of transferred $Q^*$ learning shown in the left panel of Figure \ref{['fig:chakra-online']} can be viewed as the jumpstart improvement---one of the three main objectives of transfer learning defined in langley2006transfer and taylor2009transfer. We also empirically study the cumulative regret of online $Q$-learning where the values of $\widehat{\boldsymbol{\theta}}_t$, $t=1,2$ are updated during the exploitation stage. This phase-based ETC online $Q$-learning algorithm is a natural extension of Algorithm \ref{['alg-online']} and it goes as follows. At the first phase, vanilla $Q$-learning initializes $\widehat{\boldsymbol{\theta}}_t$ to zero, while transfer $Q$-learning initializes with $\widehat{\boldsymbol{\theta}}_t$ that are estimated using offline trajectories from the source task. Then at each phase, a batch of $100$ trajectories are generated using greedy actions based on estimated $\widehat{\boldsymbol{\theta}}_t$. Using the extra generated trajectories at the current phase, the values $\widehat{\boldsymbol{\theta}}_t$ are updated and will be used to generate greedy actions for the next phase. The right panel of Figure \ref{['fig:chakra-online']} shows the mean of cumulative regret as phases proceed online. It shows all three main advantages of transferred learning, namely, jumpstart, learning speed, and asymptotic improvements langley2006transfertaylor2009transfer. Cumulative regret of online $Q$-learning with or without transfer. Left panel: The explore-then-commit (ETC) online $Q$-learning; Right panel: phase-based ETC online $Q$-learning with parameter updates. In this section, we illustrate an application of the transfer $Q$-learning in the Medical Information Mart for Intensive Care version III (MIMIC-III) Database johnson2016mimic, which is a freely available source of de-identified critical care data from 2001 -- 2012 in six ICUs at a Boston teaching hospital. We consider a cohort of sepsis patients, following the same data processing procedure as that in chen2022reinforcementkomorowski2018artificial. Each patient in the cohort is characterized by a set of 47 variables, including demographics, Elixhauser premorbid status, vital signs, and laboratory values. Patients' data were coded as multidimensional discrete time series $\boldsymbol{x}_{i,t} \in\mathbb{R}^{47}$ for $1\le i \le N$ and $1\le t \le T_i$ with 4-hour time steps. The actions of interests are the total volume of intravenous (IV) fluids and maximum dose of vasopressors administrated over each 4-hour period. We discretize two actions into three levels (i.e., low, medium, and high), respectively. In our setting, the low-level corresponds to level 1 - 2, the medium-level corresponds to level 3 and the high-level corresponds to level 4 - 5 in komorowski2018artificial. The combination of the two drugs makes $M = 3 \times 3 = 9$ possible actions in total. The final processed dataset contains 20943 unique adult ICU admissions, among which 11704 (55.88%) are female (0) and 9239 (44.11%) are male (1). The reward signal is important and is crafted carefully in real applications. For the final reward, we follow komorowski2018artificial and use hospital mortality or 90-day mortality. Specifically, when a patient survived after 90 days out of hospital, a positive reward was released at the end of each patient's trajectory; a negative reward was issued if the patient died in hospital or within 90 days out of hospital. In our dataset, the mortality rate is 24.21% for female and 22.71% for male. For the intermediate rewards, we follow prasad2017reinforcement and associates reward to the health measurement of a patient. The detailed description of the data pre-processing is presented in Section \ref{['appen:mimic-iii']} of the supplemental material. The trajectory horizons are different in the dataset, with the maximum being 20 and minimum being 1. We use 20 as the total length of horizon. The trajectories are aligned at the last steps while allowing the starting steps vary. For examples, the trajectories with length 20 start at step $1$ while the trajectories with length 10 start at step $11$. But they all end at step 20. We adopt this method because the distribution of final status are similar across trajectories. Figure \ref{['fig:mort-horizon']} presents the mortality rates of different lengths. We see that while the numbers of trajectories differ a lot, the mortality rates do not vary much across trajectories with different horizons. On the contrary, the starting status of patients may be very different. The one with trajectory length $20$ may be in a worse status and needs $10$ steps to reach the status similar to the starting status of the one with length $10$. We believe this is a reasonable setup to illustrate our method. A rigorous medical analysis is beyond the scope of this paper and is a worthwhile topic for future research. Mortality rates across different horizon lengths. We consider transfer $Q$-learning across genders. The analytical model for optimal $Q_t^{(k)}$ function for each step $t \in [20]$ is modeled by $Q_t^{(k)}\left( \boldsymbol{x}, a \right) \approx \boldsymbol{x}^\top \sum_{a'=1}^9 \boldsymbol{\theta}_{t,a}^{(k)} \boldsymbol{1}\left( a = a' \right),$ where the covariates $\boldsymbol{x}$ contains 44 variables detailed in Table \ref{['tab:variables']} in the supplements. Even though the total number of trajectories of gender $0$ is large, estimating \ref{['eqn:anal-opt-Q']} is still a high-dimensional problem since we allow $\boldsymbol{\theta}_{t,a}^{(k)}$ be different across step $t$, action $a$ and gender subgroup $k = 0,1$ and the number of trajectories corresponding to a specific combination of $t$, $a$, and $g$ is small. For example, there are only 19 samples available to estimate $\boldsymbol{\theta}_{t,a}^{(k)}$ for gender $k = 1$, step $t = 1$, and action $a$ corresponding to the combination $(\textup{IV}, \textup{Vaso}) = (0, 1)$. Table \ref{['tab:least']} in the supplemental material shows that the least ten samples sizes are all under $30$. We observe that gender $k = 1$ (male) has fewer samples so we consider transfer $Q$-learning with target task for gender $k=1$ and auxiliary source task from gender $k=0$. The estimation procedure of transfer $Q$-learning follows Algorithm \ref{['alg-master']}. We set the discount factor as $\gamma = 0.98$. We also estimate the $Q^*$ function by the vanilla $Q$-learning which is the counterpart of Algorithm \ref{['alg-master']} without transfer. The Lasso tuning parameters are chosen according to Theorem \ref{['thm1-tl']} and by a linear search for the value of $c_1$ that maximizes the objective function. We calculate the optimal aggregated values of transferred and Vanilla $Q$-learning, denoted by $V^{*,(tl)}$ and $V^{*,(st)}$, respectively. Figure \ref{['fig:optimal-value-ratio']} plot the average of the ratio $V^{*,(tl)} / V^{*,(st)}$ and its standard deviation. The mean ratio is above one, indicating that the optimal value of transfer $Q$-learning is larger. Ratio between optimal aggregated values $V^*$ of transfer $Q$-learning ($V^{*,(tl)}$) and those of Vanilla $Q$-learning ($V^{*,(st)}$) applied on the target data. Shaded area covers one standard deviation from the mean. Inverse step is $20 - t$ where $1 \le t \le 20$ is the natural step in a trajectory. In this work, we envisage $Q$-learning with knowledge transfer in the form of an algorithm that exploits the availability of samples from different but similar RL tasks. The similarity between target and source RL tasks is characterized by their corresponding reward functions which can be checked directly in practice. We note that a salient feature of RL is its multi-stage learning and, accordingly, we design a novel re-targeting step to enable "cross-stage transfer" along multiple stages in an RL task, in addition to the usual "cross-task transfer" in TL for supervised learning. For both offline and online RL tasks, our approach demonstrates improvements over single-task $Q$-learning, both theoretically and empirically. The similarity definition among multi-stage RL tasks can be diverse and more flexible. For instance, some related tasks may have different action spaces or different number of stages. It is also important to study transfer learning for such situations. As the similarity level is always unknown, it is likely that we include some source tasks which are far away from the target task. As a future research direction, one can perform a model selection aggregation step and aggregation in multi-stage models, especially the online setting, is largely an open problem and requires further study. Elynn Chen was supported by NSF Grant DMS-2412577. Bastani, HamsaH. (2021). Predicting with proxies: Transfer learning in high dimension. Management Science 67 2964--2984. Cai, T TonyT. T. Wei, HongjiH. (2021). Transfer learning for nonparametric classification: Minimax rate and adaptive classifier. The Annals of Statistics 49 100--128. Chai, JinhangJ., Chen, ElynnE. Fan, JianqingJ. (2025). Deep Transfer $Q$-Learning for Offline Non-Stationary Reinforcement Learning. arXiv preprint arXiv:2501.04870. Chai, JinhangJ., Chen, ElynnE. Yang, LinL. (2025). Transfer $Q$-Learning with Composite MDP Structures. In The Forty-second International Conference on Machine Learning (ICML 2025). Chakraborty, BibhasB., Murphy, SusanS. Strecher, VictorV. (2010). Inference for non-regular parameters in optimal dynamic treatment regimes. Statistical Methods in Medical Research 19 317--343. Chen, ElynnE., Chen, XiX. Jing, WenboW. (2024). Data-Driven Knowledge Transfer in Batch $Q$-Learning. arXiv preprint arXiv:2404.15209, 2024. Chen, Elynn YE. Y., Song, RuiR. Jordan, Michael IM. I. (2022). Reinforcement Learning in Latent Heterogeneous Environments. arXiv preprint arXiv:2202.00088. Clifton, JesseJ. Laber, EricE. (2020). Q-learning: Theory and applications. Annual Review of Statistics and Its Application 7 279--301. Ertefaie, AshkanA., McKay, James RJ. R., Oslin, DavidD. Strawderman, Robert LR. L. (2021). Robust Q-learning. Journal of the American Statistical Association 116 368--381. Ertefaie, AshkanA. Strawderman, Robert LR. L. (2018). Constructing dynamic treatment regimes over indefinite time horizons. Biometrika 105 963--977. Gretton, ArthurA., Borgwardt, Karsten MK. M., Rasch, Malte JM. J., Schölkopf, BernhardB. Smola, AlexanderA. (2012). A kernel two-sample test. The journal of machine learning research 13 723--773. Gu, TianT., Han, YiY. Duan, RuiR. (2022). Robust angle-based transfer learning in high dimensions. arXiv preprint arXiv:2210.12759. Hao, BotaoB., Lattimore, TorT., Szepesvári, CsabaC. Wang, MengdiM. (2021). Online Sparse Reinforcement Learning. In International Conference on Artificial Intelligence and Statistics 316--324. PMLR. Jin, ChiC., Yang, ZhuoranZ., Wang, ZhaoranZ. Jordan, Michael IM. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory 2137--2143. Johnson, Alistair EWA. E., Pollard, Tom JT. J., Shen, LuL., Li-wei, H LehmanH. L., Feng, MenglingM., Ghassemi, MohammadM., Moody, BenjaminB., Szolovits, PeterP., Celi, Leo AnthonyL. A. Mark, Roger GR. G. (2016). MIMIC-III, a freely accessible critical care database. Scientific Data 3 160035. Kallus, NathanN. (2020). More efficient policy learning via optimal retargeting. Journal of the American Statistical Association 1--13. Kolm, Petter NP. N. Ritter, GordonG. (2020). Modern perspectives on reinforcement learning in finance. Modern Perspectives on Reinforcement Learning in Finance (September 6, 2019). The Journal of Machine Learning in Finance 1. Komorowski, MatthieuM., Celi, Leo AL. A., Badawi, OmarO., Gordon, Anthony CA. C. Faisal, A AldoA. A. (2018). The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine 24 1716--1720. Langley, PatP. (2006). Transfer of knowledge in cognitive systems. In Talk, workshop on Structural Knowledge Transfer for Machine Learning at the Twenty-Third International Conference on Machine Learning. Lazaric, AlessandroA. (2012). Transfer in reinforcement learning: A framework and a survey. In Reinforcement Learning 143--173. Springer. Levine, SergeyS., Kumar, AviralA., Tucker, GeorgeG. Fu, JustinJ. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643. Li, SaiS., Cai, T TonyT. T. Li, HongzheH. (2022). Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology 84 149--173. Li, SaiS., Cai, T TonyT. T. Li, HongzheH. (2022). Transfer learning in large-scale gaussian graphical models with false discovery rate control. Journal of the American Statistical Association 1--13. Li, SaiS., Zhang, LinjunL., Cai, T TonyT. T. Li, HongzheH. (2023). Estimation and inference for high-dimensional generalized linear models with knowledge transfer. Journal of the American Statistical Association 1--12. Luckett, Daniel JD. J., Laber, Eric BE. B., Kahkoska, Anna RA. R., Maahs, David MD. M., Mayer-Davis, ElizabethE. Kosorok, Michael RM. R. (2019). Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the American Statistical Association. Mousavi, AminA., Nadjar Araabi, BabakB. Nili Ahmadabadi, MajidM. (2014). Context transfer in reinforcement learning using action-value functions. Computational intelligence and neuroscience 2014. Murphy, SAS. (2005). A Generalization Error for $Q$-Learning. Journal of Machine Learning Research: JMLR 6 1073--1097. Prasad, NiranjaniN., Cheng, Li FangL. F., Chivers, CoreyC., Draugelis, MichaelM. Engelhardt, Barbara EB. E. (2017). A reinforcement learning approach to weaning of mechanical ventilation in intensive care units. In 33rd Conference on Uncertainty in Artificial Intelligence, UAI 2017. Schulte, Phillip JP. J., Tsiatis, Anastasios AA. A., Laber, Eric BE. B. Davidian, MarieM. (2014). Q- and A-learning methods for estimating optimal dynamic treatment regimes. Statistical Science: A Review Journal of the Institute of Mathematical Statistics 29 640. Singla, AdishA., Rafferty, Anna NA. N., Radanovic, GoranG. Heffernan, Neil TN. T. (2021). Reinforcement Learning for Education: Opportunities and Challenges. arXiv preprint arXiv:2107.08828. Song, RuiR., Wang, WeiweiW., Zeng, DonglinD. Kosorok, Michael RM. R. (2015). Penalized $Q$-learning for dynamic treatment regimens. Statistica Sinica 25 901. Sriperumbudur, Bharath KB. K., Fukumizu, KenjiK., Gretton, ArthurA., Schölkopf, BernhardB. Lanckriet, Gert RGG. R. (2009). On integral probability metrics,$\backslash$phi-divergences and binary classification. arXiv preprint arXiv:0901.2698. Sutton, Richard SR. S. Barto, Andrew GA. G. (2018). Reinforcement Learning: An Introduction. MIT press. Taylor, Matthew EM. E. Stone, PeterP. (2009). Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research 10. Tripuraneni, NileshN., Jin, ChiC. Jordan, Michael IM. I. (2020). Provable meta-learning of linear representations. arXiv preprint arXiv:2002.11684. Yang, LinL. Wang, MengdiM. (2019). Sample-optimal parametric Q-learning using linearly additive features. In International Conference on Machine Learning 6995--7004. PMLR. Zhang, YiY., Chen, ElynnE. Yan, YujunY. (2025). Transfer Faster, Price Smarter: Minimax Dynamic Pricing under Cross-Market Preference Shift. In The 39th Conference on Neural Information Processing Systems (NeurIPS 2025), arXiv: 2505.17203. Zhou, RunlinR., Chen, ChixiangC. Chen, ElynnE. (2025). Prior-Aligned Meta-RL: Thompson Sampling with Learned Priors and Guarantees in Finite-Horizon MDPs. arXiv preprint arXiv:2510.05446. @phdthesis{mandel2017better, title={Better Education Through Improved Reinforcement Learning}, author={Mandel, Travis Scott}, year={2017} }@article{rafferty2016faster, title={Faster teaching via pomdp planning}, author={Rafferty, Anna N and Brunskill, Emma and Griffiths, Thomas L and Shafto, Patrick}, journal={Cognitive science}, volume={40}, number={6}, pages={1290--1332}, year={2016}, publisher={Wiley Online Library} }@inproceedings{reddy2017accelerating, title={Accelerating human learning with deep reinforcement learning}, author={Reddy, Siddharth and Levine, Sergey and Dragan, Anca}, booktitle={NIPS Workshop: Teaching Machines, Robots, and Humans}, year={2017} }@article{liu2022dynamic, title={Dynamic coupon targeting using batch deep reinforcement learning: An application to livestream shopping}, author={Liu, Xiao}, journal={Marketing Science}, year={2022}, publisher={INFORMS} }@article{tuncay2023reinforcement, title={A reinforcement learning based dynamic room pricing model for hotel industry}, author={Tuncay, Gamze and Kaya, Kıymet and Yılmaz, Yaren and Yaslan, Yusuf and Gündüz Öğüdücü, Şule}, journal={INFOR: Information Systems and Operational Research}, pages={1--21}, year={2023}, publisher={Taylor & Francis} }@article{bondoux2020reinforcement, title={Reinforcement learning applied to airline revenue management}, author={Bondoux, Nicolas and Nguyen, Anh Quan and Fiig, Thomas and Acuna-Agost, Rodrigo}, journal={Journal of Revenue and Pricing Management}, volume={19}, number={5}, pages={332--348}, year={2020}, publisher={Springer} }@article{chen2023reinforcement, title={A reinforcement learning approach for hotel revenue management with evidence from field experiments}, author={Chen, Ji and Xu, Yifan and Yu, Peiwen and Zhang, Jun}, journal={Journal of Operations Management}, volume={69}, number={7}, pages={1176--1201}, year={2023}, publisher={Wiley Online Library} }@article{zhou2017optimizing, title={Optimizing chemical reactions with deep reinforcement learning}, author={Zhou, Zhenpeng and Li, Xiaocheng and Zare, Richard N}, journal={ACS Central Science}, volume={3}, number={12}, pages={1337--1344}, year={2017}, publisher={ACS Publications} }@article{gottesman2019guidelines, title={Guidelines for reinforcement learning in healthcare}, author={Gottesman, Omer and Johansson, Fredrik and Komorowski, Matthieu and Faisal, Aldo and Sontag, David and Doshi-Velez, Finale and Celi, Leo Anthony}, journal={Nature medicine}, volume={25}, number={1}, pages={16--18}, year={2019}, publisher={Nature Publishing Group US New York} }@article{pollard2018eicu, author={Tom J. Pollard and Alistair E. W. Johnson and Jesse D. Raffa and Leo A. Celi and Roger G. Mark and Omar Badawi}, title={The eICU Collaborative Research Database, a freely available multi-center database for critical care research}, journal={Scientific Data}, year={2018}, day={11}, volume={5}, pages={180178}, url={https://www.nature.com/articles/sdata2018178}, doi={10.1038/sdata.2018.178}, pmid={30204154} } Supplementary Material of "Transfer $Q$-Learning for Finite-Horizon Markov Decision Processes" The supplementary material provides detailed mathematical proofs for the main theoretical results, including the convergence rate and regret bounds, together with additional lemmas and intermediate derivations. It also contains implementation details and extended results from the numerical studies, such as full simulation settings, additional tables, and figures. For the real-data application on the MIMIC-III sepsis cohort, the supplement describes the data preprocessing procedures, variable definitions, reward construction, and cohort statistics. These materials ensure full reproducibility and transparency of the theoretical and empirical findings reported in the main paper. We first define some notation. Let $\alpha_k=n_k/N_{\mathrm{src}}$ and $\bar{\boldsymbol{\Sigma}}_t=\sum_{k=1}^K\alpha_k\boldsymbol{\Sigma}^{(k)}_t$. Notice that $\mathbb{E}[\sum_{k=1}^K(\boldsymbol{W}^{(k)}_T)^\top(\boldsymbol{y}^{(k)}_T-\boldsymbol{W}^{(k)}_T\boldsymbol{b}_T)]=0$ for $\boldsymbol{b}_T=\boldsymbol{\theta}_t+\underbrace{\{\sum_{k=1}^K\alpha_k\boldsymbol{\Sigma}^{(k)}_T\}^{-1}\sum_{k=1}^K\alpha_k\boldsymbol{\Sigma}^{(k)}_T\boldsymbol{\delta}^{(k)}_T}_{\bar{\boldsymbol{\delta}}_T}.$ Define an event \mathcal{E}_t=\left\{\frac{1}{n_0}\|(\boldsymbol{W}_t^{(0)})^\top(\boldsymbol{y}_t^{(0)}-\boldsymbol{W}_t^{(0)}\boldsymbol{\theta}_t)\|_{\infty}\leq \lambda_0/2,\right.\frac{1}{N_{\mathrm{src}}}\|\sum_{k=1}^K(\boldsymbol{W}_t^{(k)})^\top(\boldsymbol{y}_t^{(tl-k)}-\boldsymbol{W}_t^{(k)}\boldsymbol{b}_t)\|_{\infty}\leq \lambda_{\mathrm{src}}/2,\quad\quad\boldsymbol{u}^\top\widehat{\boldsymbol{\Sigma}}^{(0)}_t\boldsymbol{u}\geq C\|\boldsymbol{u}\|_2^2-\|\boldsymbol{u}\|_1^2\frac{\log p}{n_0},~\boldsymbol{u}^\top\widehat{\bar{\boldsymbol{\Sigma}}}_t\boldsymbol{u}\geq C\|\boldsymbol{u}\|_2^2-\|\boldsymbol{u}\|_1^2\frac{\log p}{N_{\mathrm{src}}},\quad\quad\left. \sup_{\|\boldsymbol{u}\|_0\leq C(s+h/\sqrt{\log p/n_0})} \boldsymbol{u}^\top\widehat{\bar{\boldsymbol{\Sigma}}}_t\boldsymbol{u}\leq C\|\boldsymbol{u}\|_2^2\right\}. It is easy to show that for $s\log p/N_{\mathrm{src}}+h\sqrt{\log p/n_0}=o(1)$, $\mathbb{P}(\cap_{t=1}^T\mathcal{E}_t)\geq 1-\exp(-c_1\log p)$ for any finite $T$. Conceptually, define $\widetilde{\boldsymbol{\delta}}_T$ be a thresholded version of $\bar{\boldsymbol{\delta}}_T$ at threshold level $\sqrt{\log p/n_0}$. That is, $(\widetilde{\boldsymbol{\delta}}_T)_{j}=(\bar{\boldsymbol{\delta}}_T)_j\mathbbm{1}(|(\bar{\boldsymbol{\delta}}_T)_j|\geq \sqrt{\log p/n_0})$. We only leverage $\widetilde{\boldsymbol{\delta}}_T$ to facilitate the proof. Under the conditions of Theorem \ref{['thm1-tl']}, with probability at least $1-\exp(-c_1\log p)$ for some constant $c_1>0$, \|\widehat{\boldsymbol{\theta}}_T-\boldsymbol{\theta}_T\|_2^2\lesssim \frac{s\log p}{N_{\mathrm{src}}}+h\sqrt{\frac{\log p}{n_0}}\wedge h^2\|\widehat{\boldsymbol{\theta}}_T+(\bar{\boldsymbol{\delta}}_T-\widetilde{\boldsymbol{\delta}}_T)-\boldsymbol{\theta}_T\|_0\lesssim |S_T|\lesssim (s+h/\sqrt{\log p/n_0}). For the final stage, the analysis is analogous to the supervised linear regression. Specifically, \|\bar{\boldsymbol{\delta}}_T\|_1= \|\bar{\boldsymbol{\Sigma}}_T^{-1}\sum_{k=1}^K\alpha_k\boldsymbol{\Sigma}^{(k)}_T\boldsymbol{\delta}_T^{(k)}\|_1\leq \|\{\boldsymbol{\Sigma}_T^{(0)}\}^{-1}\sum_{k=1}^K\alpha_k\boldsymbol{\Sigma}^{(k)}_T\boldsymbol{\delta}_T^{(k)}\|_1+\|\{\boldsymbol{\Sigma}_T^{(0)}\}^{-1}\sum_{k=1}^K\alpha_k(\boldsymbol{\Sigma}^{(k)}_T-\boldsymbol{\Sigma}^{(0)}_T)\bar{\boldsymbol{\Sigma}}_T^{-1}\sum_{k=1}^K\alpha_k\boldsymbol{\Sigma}^{(k)}_T\boldsymbol{\delta}^{(k)}_T\|_1\leq (1+C_{\Sigma})\max_{k\leq K}\|\boldsymbol{\delta}^{(k)}_T\|_1+C_{\boldsymbol{\Sigma}} \|\bar{\boldsymbol{\delta}}_T\|_1. Hence, for $C_{\Sigma}<1$, $\|\bar{\boldsymbol{\delta}}_T\|_1\leq \frac{1+C_{\Sigma}}{1-C_{\Sigma}}\max_{k\leq K}\|\delta^{(k)}_T\|_1.$ Oracle inequality for $\widehat{\boldsymbol{b}}_T$: \frac{1}{N_{\mathrm{src}}}\sum_{k=1}^K\|\boldsymbol{W}_T^{(k)}(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)\|_2^2\leq \frac{1}{N_{\mathrm{src}}}|\sum_{k=1}^K\langle \boldsymbol{W}_T^{(k)}(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}), \boldsymbol{y}_T^{(tl-k)}-\boldsymbol{W}_T^{(k)}\boldsymbol{b}_T)\rangle|+\lambda_{\mathrm{src}}(\|\boldsymbol{b}_T\|_1-\|\widehat{\boldsymbol{b}}_T\|_1). Using standard arguments, in event $\mathcal{E}_T$, $\|\widehat{\bar{\boldsymbol{\Sigma}}}_T^{1/2}(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)\|_2^2\vee\|\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T\|_2^2\leq Cs\lambda_{\mathrm{src}}^2+h\lambda_{\mathrm{src}}\wedge h^2\|\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T\|_1\leq s\lambda_{\mathrm{src}}+h.$ For $\widehat{\boldsymbol{\delta}}_T$, we have the following oracle inequality \frac{1}{n_0}\|\boldsymbol{W}^{(0)}_T(\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T)\|_2^2\leq \frac{1}{n_0}\langle \boldsymbol{W}^{(0)}_T(\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T),\boldsymbol{r}_T^{(0)}-\boldsymbol{W}_T^{(0)}(\widehat{\boldsymbol{b}}_T+\bar{\boldsymbol{\delta}}_T)\rangle+\lambda_0\|\bar{\boldsymbol{\delta}}_T\|_1-\lambda_0\|\widehat{\boldsymbol{\delta}}_T\|_1\leq |\frac{1}{n_0}\langle \boldsymbol{W}^{(0)}_T(\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T),\boldsymbol{r}_T^{(0)}-\boldsymbol{W}_T^{(0)}\boldsymbol{\theta}_T\rangle|+\underbrace{(\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T)^\top\widehat{\boldsymbol{\Sigma}}^{(0)}_T(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)}_{E_{2,T}}+\lambda_0\|\bar{\boldsymbol{\delta}}_T\|_1-\lambda_0\|\widehat{\boldsymbol{\delta}}_T\|_1. For $E_{2,T}$, we have with probability at least $1-\exp(-c_1\log p)$, E_{2,T}\leq (\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T)^\top\boldsymbol{\Sigma}^{(0)}_T(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T) +|(\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T)^\top(\widehat{\boldsymbol{\Sigma}}^{(0)}_T-\boldsymbol{\Sigma}^{(0)}_T)(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)|= (\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T)^\top\boldsymbol{\Sigma}^{(0)}_T\bar{\boldsymbol{\Sigma}}_T^{-1}\bar{\boldsymbol{\Sigma}}_T(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)+|(\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T)^\top(\widehat{\boldsymbol{\Sigma}}^{(0)}_T-\boldsymbol{\Sigma}^{(0)}_T)(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)|\leq (\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T)^\top\boldsymbol{\Sigma}^{(0)}_T\bar{\boldsymbol{\Sigma}}_T^{-1}\widehat{\bar{\boldsymbol{\Sigma}}}_T(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)+|(\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T)^\top(\widehat{\boldsymbol{\Sigma}}^{(0)}_T-\boldsymbol{\Sigma}^{(0)}_T)(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)|\quad +|(\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T)^\top\boldsymbol{\Sigma}^{(0)}_T\bar{\boldsymbol{\Sigma}}_T^{-1}(\widehat{\bar{\boldsymbol{\Sigma}}}_T-\bar{\boldsymbol{\Sigma}}_T)(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)|\lesssim \|\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T\|_1 \|\boldsymbol{\Sigma}^{(0)}_T\bar{\boldsymbol{\Sigma}}_T^{-1}\|_{\infty,1}\lambda_{\mathrm{src}}+\|\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T\|_1\|\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T\|_1\sqrt{\frac{\log p}{n_0}}\quad+ \|\boldsymbol{\Sigma}^{(0)}_T\bar{\boldsymbol{\Sigma}}_T^{-1}\|_{\infty,1}\|\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T\|_1\sqrt{\frac{\log p}{N_{\mathrm{src}}}}\|\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T\|_1, where the first term comes from the KKT conditions associated with the first step, the second term comes from the sub-Gaussian nature of $\boldsymbol{w}^{(0)}_{T,i}$, and the last term comes from the sub-Gaussian nature of $\{\boldsymbol{w}^{(k)}_{T,i}\}_{k\in[K]}$. Notice that $\|\boldsymbol{\Sigma}^{(0)}_T\bar{\boldsymbol{\Sigma}}_T^{-1}\|_{\infty,1}\leq 1+\|(\boldsymbol{\Sigma}^{(0)}_T-\bar{\boldsymbol{\Sigma}}_T)\bar{\boldsymbol{\Sigma}}_T^{-1}\|_{\infty,1} \leq 1+C_{\Sigma}\|\boldsymbol{\Sigma}^{(0)}_T\bar{\boldsymbol{\Sigma}}_T^{-1}\|_{\infty,1}.$ By (\ref{['bT-l1']}), $\|\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T\|_1\leq C$ under the sample size condition of Theorem \ref{['thm1-tl']}. In event $\mathcal{E}_T$, we have $E_{2,T}\leq C'\lambda_0\|\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T\|_1$. Hence, in $\mathcal{E}_T$, we arrive at the following oracle inequality of $\widehat{\boldsymbol{\delta}}_T$: \frac{1}{n_0}\|\boldsymbol{W}^{(0)}_T(\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T)\|_2^2\leq \frac{1}{2}\lambda_0\|\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T\|_1+C\sqrt{\frac{\log p}{n_0}}\|\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T\|_1+\lambda_0\|\bar{\boldsymbol{\delta}}_T\|_1-\lambda_0\|\widehat{\boldsymbol{\delta}}_T\|_1. In $\mathcal{E}_T$, with the sample size condition and the choice of $\lambda_0$ in Lemma \ref{['lem1-tl']}, standard arguments lead to \|\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T\|_2^2\vee \frac{1}{n_0}\|\boldsymbol{W}^{(0)}_T(\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T)\|_2^2\leq Ch\sqrt{\frac{\log p}{n_0}}\|\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T\|_1\leq Ch. Notice that $P(\mathcal{E}_T)\geq \exp\{-c_1\log p\}$. To get the convergence rate of $\widehat{\boldsymbol{\theta}}_T=\check{\boldsymbol{b}}_T+\check{\boldsymbol{\delta}}_T$, we need to analyze the thresholded version of $\widehat{\boldsymbol{b}}_T$ and $\widehat{\boldsymbol{\theta}}_T$. Let $S_T=supp(\boldsymbol{\theta}_t)\cup \{j: |\bar{\boldsymbol{\delta}}_T|\geq \sqrt{\log p/n_0}\}$. Next, we show that in event $\mathcal{E}_T$, $\|\check{\boldsymbol{b}}_T-\boldsymbol{b}_T\|_2^2\lesssim_{\mathbb{P}} \frac{s\log p}{N_{\mathrm{src}}}+h\sqrt{\frac{\log p}{n_0}}\wedge h^2~\text{and}~~\|\check{\boldsymbol{b}}_T\|_0\leq C|S_T|\|\check{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T\|_2^2\leq \|\widehat{\boldsymbol{\delta}}_T-\bar{\boldsymbol{\delta}}_T\|_2^2+h\sqrt{\log p/n_0}~\text{and}~~ \|\check{\boldsymbol{\delta}}_T-\widetilde{\boldsymbol{\delta}}_T\|_0\leq C|S_T|.$ Notice that $\lambda_{\mathrm{src}}\asymp C\sqrt{\frac{\log p}{N_{\mathrm{src}}}}+\frac{h}{s\sqrt{\log p/n_0}+h}\sqrt{\frac{\log p}{n_0}}$ and $|S_T|\leq s+h/\sqrt{\log p/n_0}$. Therefore, \|\check{\boldsymbol{b}}_T-\boldsymbol{b}_T\|_2^2\leq \|(\check{\boldsymbol{b}}_T-\boldsymbol{b}_T)_{S_T}\|_2^2+\|(\check{\boldsymbol{b}}_T-\boldsymbol{b}_T)_{S_T^c}\|_2^2\leq \|(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)_{S_T}\|_2^2+\lambda_{\mathrm{src}}^2|S_T|+\|(\check{\boldsymbol{b}}_T)_{S_T^c}\|_2^2+\|(\boldsymbol{b}_T)_{S_T^c}\|_2^2\leq \|(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)_{S_T}\|_2^2+\lambda_{\mathrm{src}}^2|S_T|+\|(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)_{S_T^c}\|_2^2+2\|(\boldsymbol{b}_T)_{S_T^c}\|_2^2\leq \|\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T\|_2^2+\lambda_{\mathrm{src}}^2|S_T|+2\|(\boldsymbol{b}_T)_{S_T^c}\|_2^2\lesssim s\lambda_{\mathrm{src}}^2+h\lambda_{\mathrm{src}}\wedge h^2+(s+\frac{h}{\sqrt{\log p/n_0}})\lambda_{\mathrm{src}}^2+h\sqrt{\frac{\log p}{n_0}}\wedge h^2. Notice that $\lambda_{\mathrm{src}}=o(\sqrt{\frac{\log p}{n_0}})$ as long as $h\ll s\sqrt{\frac{\log p}{n_0}}$. Hence, \|\check{\boldsymbol{b}}_T-\boldsymbol{b}_T\|_2^2\lesssim s\lambda_{\mathrm{src}}^2+h\sqrt{\frac{\log p}{n_0}}\wedge h^2\lesssim \frac{s\log p}{N_{\mathrm{src}}}+h\sqrt{\frac{\log p}{n_0}}. Moreover, \|\check{\boldsymbol{b}}_T\|_0\leq |S_T|+\sum_{j\notin S_T} \mathbbm{1}(|(\widehat{\boldsymbol{b}}_T)_j|\geq \lambda_{\mathrm{src}})\leq |S_T|+\sum_{j\notin S_T,(\boldsymbol{b}_T)_j= 0} \mathbbm{1}(|(\widehat{\boldsymbol{b}}_T-\boldsymbol{b}_T)_j|\geq \lambda_{\mathrm{src}})+|\{j\notin S_T,(\boldsymbol{b}_T)_j\neq 0\}|\lesssim C|S_T|. For the thresholded version $\check{\boldsymbol{\delta}}_T$, it is easy to show that $\|\widetilde{\boldsymbol{\delta}}_T\|_0\leq h/\sqrt{\log p/n_0}$. The proof follows similarly. Finally, notice that \|\widehat{\boldsymbol{\theta}}_t-\boldsymbol{\theta}_t+(\bar{\boldsymbol{\delta}}_T-\widetilde{\boldsymbol{\delta}}_T)\|_0\leq \|\check{\boldsymbol{b}}_T-\boldsymbol{b}_T+\check{\boldsymbol{\delta}}_T-\widetilde{\boldsymbol{\delta}}_T\|_0\leq C|S_T|+\|\check{\boldsymbol{\delta}}_T-\widetilde{\boldsymbol{\delta}}_T\|_0\leq C'|S_T|. The proof is complete. We are left to prove the convergence rate of $\widehat{\boldsymbol{\theta}}_t-\boldsymbol{\theta}_t$ for $t<T$. For $\boldsymbol{b}_t$ defined in \ref{['eq-bt']}, the following oracle inequality holds \frac{1}{N_{\mathrm{src}}}\sum_{k=1}^K\|\boldsymbol{W}_t^{(k)}(\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t)\|_2^2\leq\frac{1}{N_{\mathrm{src}}}|\sum_{k=1}^K\langle \boldsymbol{W}_t^{(k)}(\widehat{\boldsymbol{b}}_t-\boldsymbol{b}), \widehat{\boldsymbol{y}}_t^{(k)}-\boldsymbol{W}_t^{(k)}\boldsymbol{b}_t)\rangle| +\lambda_{\mathrm{src}}(\|\boldsymbol{b}_{t}\|_1-\|\widehat{\boldsymbol{b}}_t\|_1)\leq\frac{1}{N_{\mathrm{src}}} |\sum_{k=1}^K\langle \boldsymbol{W}_t^{(k)}(\widehat{\boldsymbol{b}}_t-\boldsymbol{b}), \boldsymbol{r}_t^{(k)}+\max_{\boldsymbol{a}}Q_{t+1}(\boldsymbol{W}_{t+1}^{(k)},\boldsymbol{a};\boldsymbol{\theta}_{t+1})-\boldsymbol{W}_t^{(k)}\boldsymbol{b}_t\rangle|+\lambda_{\mathrm{src}}(\|\boldsymbol{b}_{t}\|_1-\|\widehat{\boldsymbol{b}}_t\|_1)+\underbrace{ \frac{1}{N_{\mathrm{src}}}\sum_{k=1}^K\|\max_{\boldsymbol{a}}Q_{t+1}(\boldsymbol{W}_{t+1}^{(k)},\boldsymbol{a};\widehat{\boldsymbol{\theta}}_{t+1})-\max_{\boldsymbol{a}}Q_{t+1}(\boldsymbol{W}_{t+1}^{(k)},\boldsymbol{a};\boldsymbol{\theta}_{t+1})\|_2^2}_{E_{1,t}}, where $\boldsymbol{r}_t^{(k)}+\max_{\boldsymbol{a}}Q_{t+1}(\boldsymbol{W}_{t+1}^{(k)},\boldsymbol{a};\boldsymbol{\theta}_{t+1})=\boldsymbol{y}_t^{(tl-k)}$ by definition and we use the definition of $\widehat{\boldsymbol{y}}_t^{(k)}$ and $\boldsymbol{y}_t^{(k)}$ in the last step. Using the second statement in $\mathcal{E}_t$, we have \frac{1}{N_{\mathrm{src}}}\sum_{k=1}^K\|\boldsymbol{W}_t^{(k)}(\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t)\|_2^2\leq \frac{1}{N_{\mathrm{src}}}|\sum_{k=1}^K\langle \boldsymbol{W}_t^{(k)}(\widehat{\boldsymbol{b}}_t-\boldsymbol{b}), \widehat{\boldsymbol{y}}_t^{(k)}-\boldsymbol{W}_t^{(k)}\boldsymbol{b}_t)\rangle|+\lambda_{\mathrm{src}}(\|\boldsymbol{b}_{t}\|_1-\|\widehat{\boldsymbol{b}}_t\|_1)\leq \frac{\lambda_{\mathrm{src}}}{2}\|\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t\|_1+ \lambda_{\mathrm{src}}\|\boldsymbol{b}_{t}\|_1-\lambda_{\mathrm{src}}\|\widehat{\boldsymbol{b}}_t\|_1 +E_{1,t}\leq \frac{3\lambda_{\mathrm{src}}}{2}\|(\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t)_{\text{Supp}(\boldsymbol{b}_t)}\|_1-\frac{\lambda_{\mathrm{src}}}{2}\|(\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t)_{\text{Supp}^c(\boldsymbol{b}_t)}\|_1+2\lambda_{\mathrm{src}}\|(\boldsymbol{b}_t)_{\text{Supp}^c(\boldsymbol{b}_t)}\|_1+ E_{1,t}. Similarly to the proof of Theorem 1 in li2022transfer-jrssb, we can show that under the conditions of Theorem \ref{['thm1-tl']}, if $E_{1,t}=o(1)$, then $\frac{1}{N_{\mathrm{src}}}\sum_{k=1}^K\|\boldsymbol{W}_t^{(k)}(\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t)\|_2^2\vee \|\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t\|_2^2\lesssim s\lambda_{\mathrm{src}}^2+h\lambda_{\mathrm{src}}+E_{1,t}.$ For $E_{1,t}$, we have E_{1,t}\leq \frac{1}{N_{\mathrm{src}}}\sum_{k=1}^K \|\boldsymbol{X}_{t+1}^{(k)}(\widehat{\boldsymbol{\theta}}_{t+1,1:p}-\boldsymbol{\theta}_{t+1,1:p})\|_2^2+\frac{1}{N_{\mathrm{src}}}\sum_{k=1}^K \|\boldsymbol{X}_{t+1}^{(k)}(\widehat{\boldsymbol{\theta}}_{t+1,(p+1):2p}-\boldsymbol{\theta}_{t+1,(p+1):2p})\|_2^2\leq \frac{2}{N_{\mathrm{src}}}\sum_{k=1}^K\|\boldsymbol{W}_{t+1}^{(k)}(\widehat{\boldsymbol{\theta}}_{t+1}-\boldsymbol{\theta}_{t+1})\|_2^2\leq\frac{4}{N_{\mathrm{src}}}\sum_{k=1}^K\|\boldsymbol{W}_{t+1}^{(k)}(\widehat{\boldsymbol{\theta}}_{t+1}+(\bar{\boldsymbol{\delta}}_{t+1}-\widetilde{\boldsymbol{\delta}}_{t+1})-\boldsymbol{\theta}_{t+1})\|_2^2+\frac{4}{N_{\mathrm{src}}}\sum_{k=1}^K\|\boldsymbol{W}_{t+1}^{(k)}(\bar{\boldsymbol{\delta}}_{t+1}-\widetilde{\boldsymbol{\delta}}_{t+1})\|_2^2. In the sequel, we provide the proof for $t=T-1$. The results for previous stages hold by induction. We have shown in Lemma \ref{['lem1-tl']} that $\|\widehat{\boldsymbol{\theta}}_T+(\bar{\boldsymbol{\delta}}_T-\widetilde{\boldsymbol{\delta}}_T)-\boldsymbol{\theta}_T\|_0\leq C|S_T|$. Hence, we can use the upper restricted eigenvalue condition in $\mathcal{E}_t$ to arrive at $E_{1,t}\lesssim\|\widehat{\boldsymbol{\theta}}_{T}-\boldsymbol{\theta}_T\|_2^2+\|\bar{\boldsymbol{\delta}}_T-\widetilde{\boldsymbol{\delta}}_T\|_2^2,$ with probability at least $1-\exp(-c_1\log p)$ given that $|S_T|\log p=o(N_{\mathrm{src}})$. To summarize, under the current sample size condition, with probability at least $1-\exp(-c_1\log p)$, \frac{1}{N_{\mathrm{src}}}\sum_{k=1}^K\|W_t^{(k)}(\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t)\|_2^2\vee \|\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t\|_2^2\lesssim \frac{s\log p}{N_{\mathrm{src}}}+h\sqrt{\frac{\log p}{N_{\mathrm{src}}}}+\|\widehat{\boldsymbol{\theta}}_{t+1}-\boldsymbol{\theta}_{t+1}\|_2^2\|\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t\|_1\lesssim s\lambda_{\mathrm{src}}+h+\frac{\|\widehat{\boldsymbol{\theta}}_{t+1}-\boldsymbol{\theta}_{t+1}\|_2^2}{\lambda_{\mathrm{src}}}\lesssim s\sqrt{\frac{\log p}{N_{\mathrm{src}}}}+h+\sqrt{hs}(\frac{log p}{n_0})^{1/4}. For $\widehat{\boldsymbol{\delta}}_t-\bar{\boldsymbol{\delta}}_t$, we can similarly derive that \frac{1}{n_0}\|\boldsymbol{W}^{(0)}_t(\widehat{\boldsymbol{\delta}}_t-\bar{\boldsymbol{\delta}}_t)\|_2^2\leq \frac{1}{n_0}\langle \boldsymbol{W}^{(0)}_t(\widehat{\boldsymbol{\delta}}_t-\bar{\boldsymbol{\delta}}_t),r_t^{(0)}-\boldsymbol{W}_t^{(0)}(\widehat{\boldsymbol{b}}_t+\bar{\boldsymbol{\delta}}_t)\rangle+\lambda_0\|\bar{\boldsymbol{\delta}}_t\|_1-\lambda_0\|\widehat{\boldsymbol{\delta}}_t\|_1\leq |\frac{1}{n_0}\langle \boldsymbol{W}^{(0)}_t(\widehat{\boldsymbol{\delta}}_t-\bar{\boldsymbol{\delta}}_t),r_t^{(0)}-\boldsymbol{W}_t^{(0)}\boldsymbol{\theta}_t\rangle|+\underbrace{2(\widehat{\boldsymbol{\delta}}_t-\bar{\boldsymbol{\delta}}_t)^\top\widehat{\boldsymbol{\Sigma}}^{(0)}_t(\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t)}_{E_{2,t}}+\lambda_0\|\bar{\boldsymbol{\delta}}_t\|_1-\lambda_0\|\widehat{\boldsymbol{\delta}}_t\|_1 For $E_{2,t}$, analogous to the proof of Lemma \ref{['lem1-tl']}, we have in $\mathcal{E}_t$, E_{2,t}\leq \|\widehat{\boldsymbol{\delta}}_t-\bar{\boldsymbol{\delta}}_t\|_1C\sqrt{\frac{\log p}{n_0}} (1+\|\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t\|_1). In view of (\ref{['bt-l1']}), $\|\widehat{\boldsymbol{b}}_t-\boldsymbol{b}_t\|_1\leq C$ under the sample size conditions of Theorem \ref{['thm1-tl']}. Therefore, in the event $\mathcal{E}_t$, we have \|\widehat{\boldsymbol{\delta}}_t-\bar{\boldsymbol{\delta}}_t\|_2^2\leq Ch\sqrt{\frac{\log p}{n_0}}. Let $S_t=supp(\boldsymbol{\theta}_t)\cup \{j: |\bar{\boldsymbol{\delta}}_t|\geq \sqrt{\log p/n_0}\}$. Next, we can similarly show that $\|\check{\boldsymbol{b}}_t-\boldsymbol{b}_t\|_2^2\lesssim_P \frac{s\log p}{N_{\mathrm{src}}}+h\sqrt{\frac{\log p}{n_0}}\wedge h^2~\text{and}~\|\check{\boldsymbol{b}}_t\|_0=O_P(|S_t|).$ For the thresholded version $\check{\boldsymbol{\delta}}_t$, we similarly have $\|\check{\boldsymbol{\delta}}_t-\bar{\boldsymbol{\delta}}_t\|_2^2\leq \|\widehat{\boldsymbol{\delta}}_t-\bar{\boldsymbol{\delta}}_t\|_2^2+h\sqrt{\log p/n_0}$ and $\|\check{\boldsymbol{\delta}}_t-\widetilde{\boldsymbol{\delta}}_t\|_0\leq C|S_t|.$ where $\widetilde{\boldsymbol{\delta}}_t$ is defined as a thresholded version of $\boldsymbol{\delta}_t$ at threshold level $\lambda_0$. Using the relationship between the $Q$-function and the value function, i.e., (2.1) and (2.2) in hao2021online, we have for the exploitation phase \sum_{i=1}^N\gamma^t\sum_{t=1}^Tr_t^{(0)}(\boldsymbol{x}^{(0)}_{t,i},a^*_{t,i})=\max_{a\in\{-1,1\}}Q_1(\boldsymbol{x}^{(0)}_{1,i},a)=\boldsymbol{x}^{(0)}_{1,i}\boldsymbol{\beta}_1+\gamma|\boldsymbol{x}^{(0)}_{1,i}\boldsymbol{\psi}_1|\sum_{i=n_{\mathrm{e}}+1}^N\gamma^t\sum_{t=1}^Tr_t^{(0)}(\boldsymbol{x}^{(0)}_{t,i},\widehat{a}^{(0)}_{t,i})=Q_1(\boldsymbol{x}_{1,i}^{(0)},\widehat{a}_{1,i}^{(0)})=\boldsymbol{x}^{(0)}_{1,i}\widehat{\boldsymbol{\beta}}_1+\gamma|\boldsymbol{x}^{(0)}_{1,i}\widehat{\boldsymbol{\psi}}_1| \sum_{i=1}^N\gamma^t \sum_{t=1}^T(r_t^{(0)}(\boldsymbol{x}^{(0)}_{i,t},a^*_{i,t})-r_t^{(0)}(\boldsymbol{x}^{(0)}_{i,t},\widehat{a}_{i,t}))\leq n_{\mathrm{e}}\frac{\gamma-\gamma^T}{1-\gamma}+\sum_{i=n_{\mathrm{e}}+1}^N\boldsymbol{x}_{1,i}(\boldsymbol{\beta}_1-\widehat{\boldsymbol{\beta}}_1)+\gamma(|\boldsymbol{x}^{(0)}_{1,i}\boldsymbol{\psi}_1|-|\boldsymbol{x}^{(0)}_{1,i}\widehat{\boldsymbol{\psi}}_1|)\leq \frac{n_{\mathrm{e}}\gamma}{1-\gamma}+\sum_{i=n_{\mathrm{e}}+1}^I |\boldsymbol{x}_{1,i}^{(0)}(\widehat{\boldsymbol{\theta}}_1-\boldsymbol{\theta}_1)|. As $\widehat{\boldsymbol{\theta}}_1-\boldsymbol{\theta}_1$ is independent of $\boldsymbol{x}^{(0)}_{1,i}$ for $i> n_{\mathrm{e}}$, conditioning on $\widehat{\boldsymbol{\theta}}_1-\boldsymbol{\theta}_1$, $|\boldsymbol{x}^{(0)}_{1,i}(\widehat{\boldsymbol{\psi}}_1-\boldsymbol{\psi}_1)|$ is i.i.d. sub-Gaussian with sub-Gaussian norm $C\|\widehat{\boldsymbol{\theta}}_1-\boldsymbol{\theta}_1\|_2$. Hence, \sum_{i=n_{\mathrm{e}}+1}^I |\boldsymbol{x}_{1,i}^{(0)}(\widehat{\boldsymbol{\theta}}_1-\boldsymbol{\theta}_1)|\leq (N-n_{\mathrm{e}}) \|\widehat{\boldsymbol{\theta}}_1-\boldsymbol{\theta}_1\|_2+\sqrt{N-n_{\mathrm{e}}} \|\widehat{\boldsymbol{\theta}}_1-\boldsymbol{\theta}_1\|_2, with probability at least $1-\exp\{-(N-n_{\mathrm{e}})\}$. Proof of (\ref{['reg-tl-opt']}). By (\ref{['reg-tl']}), {\rm Regret}_{NT} \lesssim \frac{n_{\mathrm{e}}\gamma }{1-\gamma} + (N-n_{\mathrm{e}}) \left( \sqrt{\frac{s\log p}{N_{\mathrm{src}}}}+ h^{1/2} \left( \frac{\log p}{n_{\mathrm{e}}} \right)^{1/4} \right):=b_N. We have b_N\lesssim n_{\mathrm{e}}+N\left( \sqrt{\frac{s\log p}{N_{\mathrm{src}}}}+ h^{1/2} \left( \frac{\log p}{n_{\mathrm{e}}} \right)^{1/4} \right). Then the optimal $n_{\mathrm{e}}$ satisfies $n_{\mathrm{e}}\asymp Nh^{1/2}(\frac{\log p}{n_{\mathrm{e}}})^{1/4}\implies n_{\mathrm{e}}\asymp (Nh^{1/2}(\log p)^{1/4})^{4/5}.$ To summarize, we see that it suffices to take $n_{\mathrm{e}} \asymp \max\{N^{4/5}h^{2/5}(\log p)^{1/5},s^2h^2\log p\}$ and the corresponding regret is of order $\max\{N^{4/5}h^{2/5}(\log p)^{1/5},s^2h^2\log p\}+N\sqrt{\frac{s\log p}{N_{\mathrm{src}}}}.$ The true coefficients for the Q-functions in (\ref{['eqn:true-q']}) are $\theta_{2j} = \kappa_j$, $1\le j\le 7$ and $\theta_{11}= \kappa_1 + q_1 \left\lvert f_1\right\rvert + q_2 \left\lvert f_2\right\rvert + (0.5-q_1) \left\lvert f_3\right\rvert + (0.5-q_2)\left\lvert f_4\right\rvert,\theta_{12}= \kappa_2 + q_1'\left\lvert f_1\right\rvert + q_2' \left\lvert f_2\right\rvert - q_1' \left\lvert f_3\right\rvert - q_2' \left\lvert f_4\right\rvert,\theta_{13}= \kappa_3 + q_1 \left\lvert f_1\right\rvert - q_2 \left\lvert f_2\right\rvert + (0.5-q_1) \left\lvert f_3\right\rvert - (0.5-q_2)\left\lvert f_4\right\rvert,\theta_{14}= \kappa_4 + q_1'\left\lvert f_1\right\rvert - q_2' \left\lvert f_2\right\rvert - q_1' \left\lvert f_3\right\rvert + q_2' \left\lvert f_4\right\rvert,$ where q_1= 0.25\left( {\rm expit}\left( b_1 + b_2 \right) + {\rm expit}\left( -b_1 + b_2 \right) \right)q_2= 0.25\left( {\rm expit}\left( b_1 - b_2 \right) + {\rm expit}\left( -b_1 - b_2 \right) \right)q_1'= 0.25\left( {\rm expit}\left( b_1 + b_2 \right) - {\rm expit}\left( -b_1 + b_2 \right) \right)q_2'= 0.25\left( {\rm expit}\left( b_1 - b_2 \right) - {\rm expit}\left( -b_1 - b_2 \right) \right)f_1= \kappa_5 + \kappa_6 + \kappa_7f_2= \kappa_5 + \kappa_6 - \kappa_7f_3= \kappa_5 - \kappa_6 + \kappa_7f_4= \kappa_5 - \kappa_6 - \kappa_7 The covariates used in real data applications are 'gender', 'age', 'elixhauser', 're_admission', 'Weight_kg','GCS', 'HR', 'SysBP', 'MeanBP', 'DiaBP', 'RR', 'SpO2', 'Temp_C', 'FiO2_1', 'Potassium', 'Sodium', 'Chloride', 'Glucose', 'BUN', 'Creatinine', 'Magnesium', 'Calcium', 'Ionised_Ca', 'CO2_mEqL', 'SGOT', 'SGPT', 'Total_bili', 'Albumin', 'Hb', 'WBC_count', 'Platelets_count', 'PTT', 'PT', 'INR', 'Arterial_pH', 'paO2', 'paCO2', 'Arterial_BE', 'Arterial_lactate', 'HCO3', 'Shock_Index', 'PaO2_FiO2', 'SOFA', 'SIRS'. The meaning of those variables are detailed in Table \ref{['tab:variables']}. Variables in MIMIC-III ItemHeaderTypeDemographicsAgeageIntegerGendergenderBinaryWeightWeight_kgContinuousReadmission to ICUre_admissionBinaryElixhauser score (premorbid status)elixhauserContinuousVital signsModified SOFASOFAContinuousSIRSSIRSContinuousGlasgow coma scaleGCSContinuousHeart rateHRContinuousSystolic blood pressureSysBPContinuousMean blood pressureMeanBPContinuousDiastolic blood pressureDiaBPContinuousShock indexShock_IndexContinuousRespiratory rateRRContinuousSpO2SpO2ContinuousTemperatureTemp_CContinuousLab valuesPotassiumPotassiumContinuousSodiumSodiumContinuousChlorideChlorideContinuousGlucoseGlucoseContinuousBUNBUNContinuousCreatinineCreatinineContinuousMagnesiumMagnesiumContinuousCalciumCalciumContinuousIonized calciumIonised_CaContinuousCarbon dioxideCO2_mEqLContinuousSGOTSGOTContinuousSGPTSGPTContinuousTotal bilirubinTotal_biliContinuousAlbuminAlbuminContinuousHemoglobinHbContinuousWhite blood cells countWBC_countContinuousPlatelets countPlatelets_countContinuousPPTPTTContinuousPTPTContinuousINRINRContinuouspHArterial_pHContinuousPaO2paO2ContinuousPaCO2paCO2ContinuousBase excessArterial_BEContinuousBicarbonateHCO3ContinuousLactateArterial_lactateContinuousPaO2/FiO2 ratioPaO2_FiO2ContinuousVentilation parametersMechanical ventilationmechventBinaryFiO2FiO2_1ContinuousMedications and fluid balanceCurrent IV fluid intake over 4hmedian_dose_vasoContinuousMaximum dose of vasopressor over 4hmax_dose_vasoContinuousUrine output over 4houtput_4hourlyContinuousCumulated fluid balance since admission (includes readmission data when available)cumulated_balanceContinuousOutcomeHospital mortalitydied_in_hospBinary90-day mortalitydied_within_48h_of_out_time,mortality_90dBinary Horizon lengths and occurrences. Length12345678910# Traces271170177184260485879114213711331Length11121314151617181920# Traces1458157315001694142910417896595244006 Table \ref{['tab:least']} presents the settings with least number of samples (top 10) in our MIMIC-III data. The settings are defined by different combinations of "Gender" $g$, "Inverse step" $T-t$ , and action "IV fluid" $\times$ "Vasopressor". Settings with least number of samples (top 10) in our MIMIC-III data. The settings are defined by different combinations of "Gender" $g$, "Inverse step" $T-t$ , and action "IV fluid" $\times$ "Vasopressor". GenderInverse stepIV fluidVaso# Samples0191118119011901901201180120119122611911261170126117112711612281181128 The data we use is the Medical Information Mart for Intensive Care version III (MIMIC-III) Database johnson2016mimic, which is a freely available source of de-identified critical care data from 53,423 adult admissions and 7,870 neonates from 2001 -- 2012 in six ICUs at a Boston teaching hospital. The database contain high-resolution patient data, including demographics, time-stamped measurements from bedside monitoring of vital signs, laboratory tests, illness severity scores, medications and procedures, fluid intakes and outputs, clinician notes and diagnostic coding. We extract a cohort of sepsis patients, following the same data processing procedure as in komorowski2018artificial. Specifically, the adult patients included in the analysis satisfy the international consensus sepsis-3 criterion. The data includes 17,083 unique ICU admissions from five separate ICUs in one tertiary teaching hospital. Patient demographics and clinical characteristics are shown in Table 1 and Supplementary Table 1 of komorowski2018artificial. Each patient in the cohort is characterized by a set of 47 variables, including demographics, Elixhauser premorbid status, vital signs, and laboratory values. Demographic information includes age, gender, weight. Vital signs include heart rate, systolic/diastolic blood pressure, respiratory rate et al. Laboratory values include glucose, total bilirubin, (partial) thromboplastin time et al. Patients' data were coded as multidimensional discrete time series with 4-hour time steps. The actions of interests are the total volume of intravenous (IV) fluids and maximum dose of vasopressors administrated over each 4-hour period. All features were checked for outliers and errors using a frequency histogram method and uni-variate statistical approaches (Tukey's method). Errors and missing values are corrected when possible. For example, conversion of temperature from Fahrenheit to Celsius degrees and capping variables to clinically plausible values. In the final processed data set, we have 17621 unique ICU admissions, corresponding to unique trajectories fed into our algorithms. For each ICU admission, we code patient's data as multivariate discrete time series with a four hour time step. Each trajectory covers from up to 24h preceding until 48h following the estimated onset of sepsis, in order to capture the early phase of its management, including initial resuscitation. The medical treatments of interest are the total volume of intravenous fluids and maximum dose of vasopressors administered over each four hour period. We use a time-limited parameter specific sample-and-hold approach to address the problem of missing or irregularly sampled data. The remaining missing data were interpolated in MIMIC-III using multivariate nearest-neighbor imputation. After processing, we have in total 278598 sampled data points for the entire sepsis cohort. The state $\boldsymbol{X}_{i,t}$ is a 47-dimensional feature vector including fixed demographic information (age, weight, gender, admit type, ethnicity et al), vitals signs (heart rate, systolic/diastolic blood pressure, respiratory rate et al), and laboratory values (glucose, Creatinine, total bilirubin, partial thromboplastin time, $paO_2$, $paCO_2$ et al.). For action space, we discretize two variables into three actions respectively according to Table in komorowski2018artificial. The combination of the two drugs makes $3 \times 3 = 9$ possible actions in total. The action $A_t$ is a two-dimensional vector, of which the first entry $a_t[0]$ specifies the dosages of IV fluids and the second $a_t[1]$ indicates the dosages of IV fluids and vasopressors, to be administrated over the next 4h interval. The reward signal is important and needs to be crafted carefully in real applications. komorowski2018artificial uses hospital mortality or 90-day mortality as the sole defining factor for the penalty and reward. Specifically, when a patient survived, a positive reward was released at the end of each patient's trajectory (a reward of + 100); while a negative reward (a penalty of -100) was issued if the patient died. However, this reward design is sparse and provides little information at each step. Also, mortality may be correlated with the health statues of a patient. So it is reasonable to associate reward to the health measurement of a patient after an action is taken. In this application, we build our reward signal based on physiological stability. Specifically, in our design, physiological stability is measured by vitals and laboratory values $v_t$ with desired ranges $[ v_{\min}, v_{\max} ]$. Important variables related to sepsis include heart rate (HR), systolic blood pressure (SysBP), mean blood pressure (MeanBP), diastolic blood pressure (DiaBP), respiratory rate (RR), peripheral capillary oxygen saturation (SpO2), arterial lactate, creatinine, total bilirubin, glucose, white blood cell count, platelets count, (partial) thromboplastin time (PTT), and International Normalized Ratio (INR). We encode a penalty for exceeding desired ranges at each time step by a truncated Sigmoid function, as well as a penalty for sharp changes in consecutive measurements. Here, values $v_t$ are the measurements of those vitals $v$ believed to be indicative of physiological stability at time $t$, with desired ranges $[v_{min}, v_{max}]$. The penalty for exceeding these ranges at each time step is given by a truncated sigmoid function. The system also receives negative feedback when consecutive measurements see a sharp change. There are definitely improvements in shaping the reward space. For example, in medical situation, the definition of the normal range of a variable sometime depends demographic characterization. Also, sharp changes in a favorable direction should be rewarded.