Validation of Composite Systems by Discrepancy Propagation

TL;DR

This work presents a validation method that propagates bounds on distributional discrepancy measures through a composite system, thereby allowing us to derive an upper bound on the failure probability of the real system from potentially inaccurate simulations.

Abstract

Assessing the validity of a real-world system with respect to given quality criteria is a common yet costly task in industrial applications due to the vast number of required real-world tests. Validating such systems by means of simulation offers a promising and less expensive alternative, but requires an assessment of the simulation accuracy and therefore end-to-end measurements. Additionally, covariate shifts between simulations and actual usage can cause difficulties for estimating the reliability of such systems. In this work, we present a validation method that propagates bounds on distributional discrepancy measures through a composite system, thereby allowing us to derive an upper bound on the failure probability of the real system from potentially inaccurate simulations. Each propagation step entails an optimization problem, where -- for measures such as maximum mean discrepancy (MMD) -- we develop tight convex relaxations based on semidefinite programs. We demonstrate that our propagation method yields valid and useful bounds for composite systems exhibiting a variety of realistic effects. In particular, we show that the proposed method can successfully account for data shifts within the experimental design as well as model inaccuracies within the simulation.

Figures (6)

• Figure 1: Illustration of our validation task: A real, composite system of interest (top) is modeled with corresponding simulation models (bottom). Measurements of the real system are available only for the individual components, while end-to-end simulation data can be generated from the models. The task of the virtual validation method is to estimate the real system performance $S$ based on the simulations $M$, incorporating simulation model misfits w.r.t. the real-world components as well as any data-shift between the simulation input distribution and the field usage to be expected in the real system.
• Figure 2: Illustration of the (marginals of the) joint input-output distribution $p_\alpha$ (\ref{['eq:p-alpha-joint']}), parameterized by weights $\alpha_v$. Corresponding in-/outputs $x_v,y_v$ have the same weight $\alpha_v$.
• Figure 3: Illustration of DPBound for a linear mapping between (samples from) Gaussian signals. (a) There is model mismatch $M\neq S$, but the input distribution $q_x=p_x$ is perfect. (b)$M=S$ is a perfect model, but the model input distribution $q_x\neq p_x$ is biased w.r.t. the real world. The computed weights $\alpha_v$ from Eqs. (\ref{['eq:p-alpha-joint']}),(\ref{['max-discrepancy-objective-alpha']}) are depicted by the size of the blue $S(x)$-markers ($\alpha$ is uniform in case (a)).
• Figure 4: Illustration of DPBound (see also Fig. \ref{['fig:linear_use_case_illustration']} in the main text) and SurrModel for a linear mapping between Gaussian signals. (a) Model and system are different $S\neq M$, whereas the input distributions are identical $p_x=q_x$. (b) The model $M$ is the perfect model, i.e. $S=M$, but input distributions are different. Computed weights $\alpha_v$ (see Eq. (\ref{['eq:p-alpha-joint']}) in Sec. \ref{['subsec:validation-method']}) are indicated by the size of markers for $S(x)$ and the worst-case distributions w.r.t. the failure probability are indicated in red. The inputs and outputs of the surrogate model SurrModel are shown in pink.
• Figure 5: Figure showing the input/output signal distributions for the "Chained Solvers" use-case in the setting "Biased Input--Misfit Model" (cf. main Tab. \ref{['tab:results_benchmark']}; we chose this use-case for illustration purposes, as its signals are one-dimensional). Top row: ground-truth signals (from the system $S$). Middle row: simulation signals (from the model $M$). Bottom row: surrogate model signals (from the model $M'$ in the SurrModel method, see Sec. \ref{['sec:uncertainty-wrapper-method']}). Left column: input distributions. Middle column: output distributions after first component. Right column: final TPI distributions.

Theorems & Definitions (5)

• Proposition 1
• Lemma 1: Tightened SDP relaxation of (\ref{['eq-no-triang:app-first-term-objective']}--\ref{['eq-no-traing:app-first-term-last-constraint']})
• proof : Proof of Prop. \ref{['prop:convergence']}
• Remark 1: Upper bounds in the limit
• Remark 2: Arbitrary simulation