Dynamical Response of the Kitaev Spin Liquid under Third-Nearest-Neighbor Heisenberg Interaction

Abstract

Motivated by growing evidence for the significance of the third-nearest-neighbor Heisenberg ($J_3$) interaction in candidate Kitaev materials, we investigate the dynamical properties of the Kitaev spin liquid (KSL) under a $J_3$ perturbation, focusing on its spin dynamical structure factor (DSF) and Raman scattering. Within a self-consistent parton mean-field plus random-phase approximation framework, we find that $J_3$ induces coherent, paramagnon-like collective modes that coexist with a high-energy Majorana continuum in the spin DSF. The softening of these modes with increasing $|J_3|$ signals a quantum phase transition to magnetic order. Remarkably, magnetic ordering sets in at a common critical $J_3$ for both ferromagnetic ($K<0$) and antiferromagnetic ($K>0$) Kitaev models, with the resulting ordered states forming exact dual pairs under a four-sublattice duality transformation that maps $(K,J_3) \rightarrow (-K,J_3)$. An external magnetic field further softens the preexisting paramagnon modes, thereby enhancing magnetic order. Perturbative Raman calculations show that while the Kitaev-like Raman vertex probes only itinerant matter Majorana fermions, the response from the $J_3$-like vertex features both matter Majoranas and visons. Four-vison excitations produce a sharp peak accompanied by a two-fermion continuum, whereas two-vison excitations yield a continuum closely resembling the single-matter-fermion density of states. These results provide a unified perspective on the dynamical signatures of $J_3$-perturbed KSL and are helpful for interpreting experimental spectra in candidate Kitaev materials with sizable $J_3$ interactions.

Figures (11)

• Figure 1: (a) Schematic of the honeycomb lattice and spin axes. The A and B sublattices are indicated by black and white dots, respectively. The gray shaded region indicates a unit cell. The $\alpha$-type bonds are aligned parallel to $\bm{\delta}_\alpha$. The spin axes are illustrated by blue arrows labeled $\mathbf{e}_\alpha$. The crystal axes $\mathbf{a}$, $\mathbf{b}$, and $\mathbf{c}$ are also shown. The dashed line indicates the mirror plane associated with the transformation $\mathcal{M}_b$. (b) Brillouin zone and high-symmetry $k$ points. The red contour indicates the momentum path used to plot the spin DSF. (c) Magnetic orders induced by the $J_3$ interaction in both the FM ($K=-1$) and AFM ($K=1$) Kitaev models. For the FM Kitaev interaction, zigzag order emerges when $J_3 \ge 0.094$, while for $J_3 \le -0.094$, the soft modes indicate competing tendencies toward FM and stripe order (FM+stripe). For the AFM Kitaev interaction, stripe order emerges when $J_3 \le -0.094$, whereas for $J_3 \ge 0.094$, the soft modes suggest competing tendencies toward Néel AFM and zigzag order (AFM+zigzag). The critical values of $J_3$ coincide in the two models due to the $\mathcal{T}_4$ duality transformation $K \rightarrow -K$, under which the corresponding states are related. Note that the zigzag (stripe) configuration illustrated here is shown with equal spin-$x$ and spin-$y$ components as an example; under the $\mathcal{T}_4$ duality, this maps to a state with spin $x$ AFM (FM) and spin $y$ zigzag (stripe) components. This example is intended solely to illustrate the mapping and does not imply that the corresponding phase is necessarily a multi-$\mathbf{Q}$ state. The spin $z$ component is set to zero based on energetic considerations.
• Figure 2: Mean-field fermion band dispersions at selected $J_3$ for an FM Kitaev interaction ($K=-1$). (a) Fermion bands for the pure FM Kitaev model ($J_3 = 0$). The gauge fermions are static and form degenerate flat bands. (b) A finite $J_3$ term ($J_3 = 0.04$) induces gauge fluctuations, rendering the gauge-fermion bands dispersive. (c) The gauge-fermion bands become more dispersive at larger $J_3 = 0.092$. (d) Fermion bands at $J_3 = 0.092$ in the presence of a magnetic field $\mathbf{h}=0.2\, \mathbf{a}$. Hybridization between gauge and matter fermions further enhances the band dispersion and gaps out the Dirac cone at the $\mathrm{K}$ point.
• Figure 3: Mean-field [(a), (c)] versus RPA-corrected [(b), (d)] spin DSFs: (a)--(b) for the ferromagnetic Kitaev model $(K,J_3)=(-1,0)$; (c)--(d) for $(K,J_3)=(-1,0.04)$. The structure factors are plotted on a logarithmic scale, $\ln[1+S(\omega,\mathbf{q})]$.
• Figure 4: RPA spin DSFs in different regimes of $(K,J_3)$. For the AFM Kitaev model, the gap closing of the $\mathrm{M}'$ and $\mathrm{M}$ modes at (a) $J_3 = -0.094$ signals a transition to the stripe order, while the gap closing of the $\Gamma'$ and $\mathrm{M}$ modes at (b) $J_3 = 0.094$ indicates the emergence of the AFM+zigzag regime. For the FM Kitaev model, at (c) $J_3 = -0.094$, the gap closing of a sharp $\Gamma$ mode together with additional closings at $\Gamma'$, $\mathrm{M}'$, and $\mathrm{M}$ signals a transition to the FM+stripe regime, whereas the "condensation" of the $\mathrm{M}$ mode at (d) $J_3 = 0.094$ indicates a transition to the zigzag order.
• Figure 5: Spin DSF for $(K,J_3) = (-1,0.092)$ in the presence of a finite external magnetic field. For fields applied along each of the three crystallographic directions, the $\mathrm{M}$ mode softens under the magnetic field, signaling a field-induced transition to the zigzag order.