PCIe400 generic readout board qualification test

TL;DR

The PCIe400 board addresses the need for a high-throughput, flexible readout platform capable of interfacing up to 48 bidirectional links with 400 Gbit/s back-end bandwidth and phase-deterministic clock distribution for particle physics experiments. The authors combine board-level validation, BER testing, and phase-determinism measurements using an Agilex 7 M-series FPGA, PCIe Gen5, and a PAM4-based 4x100 Gbit/s interface, supported by a comprehensive software framework. Key results show BER better than $10^{-15}$ for PCIe and 4x100 Gbit/s paths and sub-$10\,\mathrm{ps}$ phase determinism across channels, with TX reset stability insights guiding mitigation strategies. The work demonstrates the PCIe400's viability for LHCb Upgrade II and similar environments, providing a reusable validation toolkit and a pathway toward full feature qualification, including clock distribution across gateware, temperature, and system-level variations.

Abstract

The PCIe400 is a generic board for high-throughput data acquisition systems in high energy physics experiments. Its purpose is to interface up to 48 bidirectional links, supporting custom protocols at 1 to 26 Gbit/s, to modern commercial back-end links providing 400 Gbit/s bandwidth. It also targets clock distribution with phase determinism below 10 ps peak-to-peak. It has been designed for LHCb LS3 enhancement upgrade with experimental features to prepare LHCb Upgrade II, foreseeing an aggregated throughput of 200 Tbit/s. However, its versatility allows it to be used in several experimental environments. The board embeds Altera's flagship Agilex 7 M-series FPGA with a PCIe Gen 5 interface and an experimental QSFP112 serial interface. We present the results of qualification tests performed on prototype boards and the challenges encountered to meet specifications. Section 1 describes board-level validation, including power-up behavior and peripheral access. Section 2 focuses on high-bandwidth interface qualification through BER measurements. Finally, Section 3 investigates phase determinism in Agilex transceivers, a key requirement for precise clock distribution.

Figures (8)

• Figure 1: (left) Block diagram of the PCIe400 board highlighting the FPGA as the central element interfacing PCIe, 400GbE and custom high-speed links. (right) Top view of the PCIe400 prototype illustrating the high component integration density.
• Figure 2: Software architecture of the PCIe400, showing the layered organization used for functional validation, monitoring, and interactive debugging.
• Figure 3: PCIe tests performed on a PCIe Gen 4 server in 2×8 bifurcated mode, demonstrating correct enumeration and DMA transfers with a BER below $10^{-15}$ at 95% confidence level on both interfaces.
• Figure 4: PRBS-31 bit-error-rate measurement on the 4x100 Gbit/s interface. Seven hours of continuous testing show no uncorrected errors, corresponding to a BER below $1\cdot10^{-15}$ at 95% confidence level.
• Figure 5: Test setup implemented on the development kit to study transceiver phase behavior. Four serial channels operate at 10.24 Gbit/s, with phase differences measured internally using DDMTD.