Trans-dimensional Hamiltonian model selection and parameter estimation from sparse, noisy data

Abstract

High-throughput characterization often requires estimating parameters and model dimension from experimental data of limited quantity and quality. Such data may result in an ill-posed inverse problem, where multiple sets of parameters and model dimensions are consistent with available data. This ill-posed regime may render traditional machine learning and deterministic methods unreliable or intractable, particularly in high-dimensional, nonlinear, and mixed continuous and discrete parameter spaces. To address these challenges, we present a Bayesian framework that hybridizes several Markov chain Monte Carlo (MCMC) sampling techniques to estimate both parameters and model dimension from sparse, noisy data. By integrating sampling for mixed continuous and discrete parameter spaces, reversible-jump MCMC to estimate model dimension, and parallel tempering to accelerate exploration of complex posteriors, our approach enables principled parameter estimation and model selection in data-limited regimes. We apply our framework to a specific ill-posed problem in quantum information science: recovering the locations and hyperfine couplings of nuclear spins surrounding a spin-defect in a semiconductor from sparse, noisy coherence data. We show that a hybridized MCMC method can recover meaningful posterior distributions over physical parameters using an order of magnitude less data than existing approaches, and we validate our results on experimental measurements. More generally, our work provides a flexible, extensible strategy for solving a broad class of ill-posed inverse problems under realistic experimental constraints.

Figures (5)

• Figure 1: Schematic of workflow. The input (left) to the hybrid MCMC algorithm is a fixed set of data $\mathbf{d}_d \in \mathbb{R}^d$, a family of candidate Hamiltonians $\{ \mathcal{H}_{k_1}, \mathcal{H}_{k_2}, \dots, \mathcal{H}_{k_j}\}$ which are parameterized by $\mathbf{a}_{k_i}$ where the number of parameters for different Hamiltonians $\mathcal{H}_{k_i}$ and $\mathcal{H}_{k_j}$ can be different. We also take in a forward model $f_k(\mathcal{H}_{k}(\mathbf{a}_k), \mathbf{b}, \mathbf{c})$ that generates a set of data from candidate Hamiltonian $\mathcal{H}_k$, and a likelihood model $\mathcal{L}(\mathbf{d}_d | f_k(\mathcal{H}_{k}(\mathbf{a}_k), \mathbf{b}, \mathbf{c}))$ that quantifies the probability of observing the data $\mathbf{d}_d$ given a set of model parameters, and any priors, which may come in the form of ab initio data. The output for the hybrid MCMC algorithm is a probability distribution over the different Hamiltonian models, and for each Hamiltonian model we have a posterior distribution over the values of the parameters.
• Figure 2: Schematic of algorithms (a) random walk Metropolis Hastings in over a continuous domain (Alg. \ref{['alg:rwmh']}), (b) random walk Metropolis Hastings over a discrete domain (Alg. \ref{['alg:rwmh']}), (c) reverse jump Markov chain Monte Carlo (Alg. \ref{['alg:rjmcmc']}), and (d) parallel tempering (Alg. \ref{['alg:parallel_temp']}).
• Figure 3: Recovery in sparse data limit. (a) We simulate a system of ten $^{13}$C surrounding a nitrogen vacancy (NV) center in diamond, and plot the resulting coherence signal for a 16-pulse dynamical decoupling experiment in an external magnetic field of 311 G. We sample a varying number of data points with noise $\epsilon \sim \mathcal{N}(0, 0.001)$, and we plot the coherence signal from the nuclear spin parameters recovered from the specified number of sampled data points. Posterior distribution of the hyperfine components of the nuclear spins (b) and relative decoherence parameter ($\lambda$) (c) recovered from the coherence signal with varying number of data points sampled corresponding to the coherence signal in $\textbf{(a)}$. In (d), (e), and (f), we plot the average results for detection rate of hyperfine couplings, the discrepancy the number of recovered spins in the bath with the number of spins that were simulated, and false positive rate for 16 simulated nuclear spin configurations containing between 5-20 nuclear spins, with the rates plotted by hyperfine magnitude ($\sqrt{A_{\perp}^2 + A_{\parallel}^2}$) of the spins.
• Figure 4: Recovery in noisy data limit. (a) We simulate a system of seventeen $^{13}$C surrounding a nitrogen vacancy (NV) center in diamond, and plot the resulting coherence signal sampled uniformly for 250 $\tau_j$ for a 16-pulse dynamical decoupling experiment in an external magnetic field of 311 G. We add varying amounts of noise to each data point, and we plot the coherence signal from the nuclear spin parameters recovered from the specified number of sampled data points. We plot the posterior distribution of the hyperfine components of the nuclear spins (b) and relative decoherence parameter ($\lambda$) (c) recovered from the coherence signal with varying number of data points sampled corresponding to the coherence signal in $\textbf{(a)}$. In (d), (e), and (f), we plot the average results for detection rate of hyperfine couplings, the discrepancy the number of recovered spins in the bath with the number of spins that were simulated, and false positive rate for 16 simulated nuclear spin configurations containing between 5-20 nuclear spins, with the rates plotted by hyperfine magnitude ($\sqrt{A_{\perp}^2 + A_{\parallel}^2}$) of the spins.
• Figure 5: Validation of hybrid MCMC approach on experimental data: (a) Average error per data point as a function of the number of MCMC steps, with five random ensembles initialized for 25,000 steps each; the first 10,000 steps are treated as burn-in, and the remaining 15,000 steps are the posterior spin configurations. (b) Experimental coherence data (250 data points from 6 $\mu$s to 8 $\mu$s of a 32-pulse CPMG sequence) to which we applied the hybrid MCMC algorithm, and the reconstructed signal using the best-fit spin configuration from the posterior distribution. (c) Posterior distribution of the total number of spins in the recovered spin configurations. (d) Posterior distribution of hyperfine couplings, with the 50 reported nuclear spin hyperfine couplings from the reference study overlaid. (e) Posterior distribution of the number of spins by spin magnitude ($\sqrt{A_\perp^2 + A_\parallel^2}$), with the mode displayed and the reference value reported in parentheses.