High-speed quantum key distribution (QKD) systems have achieved repetition frequencies above gigahertz through advanced technologies and devices, laying an important foundation for the deployment of high-key-rate QKD system. Although these advanced systems may introduce potential loopholes, an eavesdropper Eve is challenging to exploit them by performing the intercept-resend attacks due to the limited time window under high repetition frequency. However, here, we propose a security analysis model of orthogonal state attacks that do not require intercept-resend operation on the key rate of a QKD system. Under this framework, we propose a muted attack and experimentally verify the feasibility of the attack using a 1 GHz single-photon avalanche detector (SPAD). By sending hundreds of photons each time, Eve can mute Bob's SPADs to control the overall detection response of the QKD receiver, allowing her to learn nearly all the keys. Furthermore, we use this security model to simulate the overestimated key rates of the QKD system under orthogonal state attacks, including both the muted attack and the dead-time attack. This work theoretically and experimentally shows a timely case of the security vulnerability in the high-speed QKD system.