The Spectroscopic Data Processing Pipeline for the Dark Energy Spectroscopic Instrument

TL;DR

The DESI spectroscopic data pipeline converts raw CCD images from 30 cameras into calibrated spectra and redshifts for tens of millions of targets. It achieves this with a forward-model 2D PSF spectral extraction, rigorous CCD calibration, precise wavelength calibration, robust sky subtraction via PCA, and a comprehensive flux calibration anchored to standard stars and white dwarfs. The pipeline delivers high redshift efficiency and purity across target classes, supported by extensive real-time QA and a scalable, open-source software ecosystem. This framework enables DESI to meet its ambitious goals for cosmology while providing well-documented data products for public release and community use.

Abstract

We describe the spectroscopic data processing pipeline of the Dark Energy Spectroscopic Instrument (DESI), which is conducting a redshift survey of about 40 million galaxies and quasars using a purpose-built instrument on the 4-m Mayall Telescope at Kitt Peak National Observatory. The main goal of DESI is to measure with unprecedented precision the expansion history of the Universe with the Baryon Acoustic Oscillation technique and the growth rate of structure with Redshift Space Distortions. Ten spectrographs with three cameras each disperse the light from 5000 fibers onto 30 CCDs, covering the near UV to near infrared (3600 to 9800 Angstrom) with a spectral resolution ranging from 2000 to 5000. The DESI data pipeline generates wavelength- and flux-calibrated spectra of all the targets, along with spectroscopic classifications and redshift measurements. Fully processed data from each night are typically available to the DESI collaboration the following morning. We give details about the pipeline's algorithms, and provide performance results on the stability of the optics, the quality of the sky background subtraction, and the precision and accuracy of the instrumental calibration. This pipeline has been used to process the DESI Survey Validation data set, and has exceeded the project's requirements for redshift performance, with high efficiency and a purity greater than 99 percent for all target classes.

Figures (55)

• Figure 1: Schematic view of one of the 10 DESI spectrographs. See text for details.
• Figure 2: DESI CCD image layout with four amplifiers. The CCD images for all cameras have the same orientation after transformation of the data array by the instrument control system; the column index (AXIS1 in the fits files headers, often labeled $X$ in this paper) increases with the fiber number, and the row index (AXIS2 in fits, often label $Y$ in this paper) increases with increasing wavelength. The black arrows indicate the direction of readout, with the parallel clock in the wavelength direction ($Y$) and the serial clock along the fiber number ($X$). Also shown are the prescan and overscan regions used to measure and remove the bias level during pre-processing (see §\ref{['sec:preprocessing']}).
• Figure 3: Illustration of the DESI calibration system with the location of the screen and calibration lamps. The diameter of the dome screen is 5 m.
• Figure 4: Example CCD image after preprocessing. This is a NIR CCD image of spectrograph SM10 after a 900s exposure. On the top panel, one can see the 500 fibers organized in 20 blocks of 25 fibers each. The mostly horizontal curved lines are sky lines. Bright fibers (appearing as vertical dark bands in this negative color scale) are the spectral traces from standard star fibers. One can also note many cosmic ray hits. The lower panel is a zoom highlighting the clear separation of the fiber traces, the space between blocks, and the spectrograph resolution (see §\ref{['sec:psf']} for more details on the resolution).
• Figure 5: Spectroscopic pipeline data flow