Optimal and Unbiased Fluxes from Up-the-Ramp Detectors under Variable Illumination

TL;DR

This work addresses flux extraction from NIR up-the-ramp detectors under time-variable illumination caused by changing atmospheric conditions. It develops a Gaussian-difference, Poisson-Gaussian noise model that incorporates a shared time-variation factor $\vec{b}$ across neighboring pixels and solves for pixel fluxes $f_p$ using a linearized MAP approach, with a degeneracy-fixed constraint on $\vec{b}$. Synthetic experiments show that the time-variable model yields unbiased fluxes with near-optimal uncertainties, while neglecting variability induces strong, flux-dependent biases; real APOGEE data corroborate achromatic, atmospheric-driven variability and demonstrate improved data-model agreement when using the new model. The results suggest adopting the time-variable approach whenever the amplitude of flux change exceeds about $3.5\%$, and plan to integrate this method into APOGEE data reduction pipelines for robust, scalable corrections to time-varying NIR observations.

Abstract

Near-infrared (NIR) detectors -- which use non-destructive readouts to measure time-series counts-per-pixel -- play a crucial role in modern astrophysics. Standard NIR flux extraction techniques were developed for space-based observations and assume that source fluxes are constant over an observation. However, ground-based telescopes often see short-timescale atmospheric variations that can dramatically change the number of photons arriving at a pixel. This work presents a new statistical model that shares information between neighboring spectral pixels to characterize time-variable observations and extract unbiased fluxes with optimal uncertainties. We generate realistic synthetic data using a variety of flux and amplitude-of-time-variability conditions to confirm that our model recovers unbiased and optimal estimates of both the true flux and the time-variable signal. We find that the time-variable model should be favored over a constant-flux model when the observed count rates change by more than 3.5%. Ignoring time variability in the data can result in flux-dependent, unknown-sign biases that are as large as ~120% of the flux uncertainty. Using real APOGEE spectra, we find empirical evidence for approximately wavelength-independent, time-dependent variations in count rates with amplitudes much greater than the 3.5% threshold. Our model can robustly measure and remove the time-dependence in real data, improving the quality of data-model comparison. We show several examples where the observed time-dependence quantitatively agrees with independent measurements of observing conditions, such as variable cloud cover and seeing.

Figures (20)

• Figure 1: Top: An example "ramp" of counts versus read number using synthetic data. These data are generated using a Poisson distribution to simulate photon arrival with Gaussian read noise, as described in the beginning of Section \ref{['sec:statistics']}. The blue line represents the observed counts in a pixel with constant flux over time, while the orange line shows the observed counts for flux that changes during the exposure to simulate the effects of, for example, a cloud moving in between reads six and seven. Bottom: Count differences versus time using the same data as in the top panel. The true input flux for both cases is 3000 electrons per read, and the orange line's flux is decreased to half of its maximum after six reads.
• Figure 2: Comparison of the best-fit $\vec{b}$ and $(\vec{f} \mid \vec{b}= \vec{b}_{\mathrm{best}})$ and the truth for synthetic data from 100 pixels in a photon-dominated regime. The left panels focus on $\vec{b}$, and the right panels focus on $(\vec{f} \mid \vec{b})$. The top panels compare the input and outputs directly, while the bottom panels show the uncertainty-scaled residuals. The bottom panels display the best-fit $\chi^2$ scaled by the appropriate degrees of freedom, both of which are near 1.0 as expected for reasonable measurements and uncertainties. The red points in the $\vec{b}$ panels mark the read time that was not used in the fitting to break degeneracies between $\vec{b}$ and $\vec{f}$, as discussed in Section \ref{['sec:statistics']}.
• Figure 3: Comparison of the data $\chi^2$ empirical cumulative distribution function (CDF) for the same simulated data as in Figure \ref{['fig:synthetic_single_output_comp']}. The blue line shows the $\chi^2$, as calculated using Equation \ref{['eq:chi2']} when using the best fit $\vec{f}$ and $\vec{b}$, which shows good agreement with the expected $\chi^2_{\nu = n_\mathrm{reads}-2}$ distribution (black line). The orange line shows the data $\chi^2$ after measuring the best-fit constant flux following the Brandt_2024 model, which is not able to describe the data within its uncertainties.
• Figure 4: Comparison between the time-independent (orange lines) versus time-variable (blue lines) versus equal-weight (green lines) models for the extracted flux uncertainties (top), median biases (middle), and residual distribution widths (bottom). Black lines and shaded regions correspond to the expected values and scatter given independent draws from a unit Gaussian. These measurements come from 1000 simulations of data using the same $\vec{b}$ as in Figure \ref{['fig:synthetic_single_output_comp']}, but using a large range of flux levels. The top panel shows that time-variable and time-independent models predict extremely similar flux uncertainties (differences much smaller than 1%) while the equal-weight model predicts larger uncertainties, especially at the high flux end. The middle panel emphasizes that neglecting to model time variability leads to a substantial flux-dependent bias when using the Brandt_2024 approach. The bottom panel shows that all the models have residual distributions with the expected widths.
• Figure 5: Summary of $\vec{b}$ measurements from repeated simulation across a range of flux levels using $n_\mathrm{reads} = 33$ and $n_\mathrm{pixels} = 100$. Each data point comes from 1000 iterations of fitting data with the same true $\vec{b}$ as shown in Figure \ref{['fig:synthetic_single_output_comp']}. The top panel shows the median uncertainty in the measured $\vec{b}$ from all realizations. The middle panel shows the 16-th, 50-th, and 84-th percentiles of the $\chi^2$ distribution of measured $\vec{b}$ compared to the truth. The bottom panel shows the median of the $\vec{b}$ uncertainty-scaled residual distribution. This figure reveals that flux levels $>10~e^-$ are able to measure unbiased $\vec{b}$ using 100 pixels.