Kitaev Meets AKLT: Competing Quantum Disorder in Spin-3/2 Honeycomb Systems

TL;DR

This work addresses how two qualitatively distinct quantum disordered states—the Kitaev QSL and the AKLT VBS—compete in a spin-$\tfrac{3}{2}$ honeycomb model that interpolates between these limits via $\xi$. Using three complementary approaches—classical O(3) vectors, semi-classical SU(4) coherent states, and exact diagonalization—the authors map the ground-state phase diagram and quantify the impact of quantum fluctuations across $\xi$. They find that classically stable noncoplanar orders are progressively destabilized by quantum fluctuations, giving way to a quantum-entangled, spin-correlations-suppressed state in the competing regime; ED reveals large entanglement entropies and weak or absent magnetic order, with signatures near Kitaev limits. Anisotropy studies further reveal inclination toward zigzag and stripy correlations, highlighting how higher-order AKLT interactions and bond anisotropy shape the quantum phases. Overall, the results illuminate the delicate balance between competing quantum disordered states in higher-spin Kitaev–AKLT systems and point to potential realizations in $t_{2g}$-based materials.

Abstract

We investigate an S=3/2 quantum spin model on a two-dimensional honeycomb lattice that continuously interpolates between two paradigmatic quantum disordered states with distinct entanglement structures: the Kitaev quantum spin liquid and the Affleck-Kennedy-Lieb-Tasaki (AKLT) valence bond solid. Combining classical, semi-classical, and exact diagonalization approaches, we map out the ground-state phase diagram and elucidate the role of quantum fluctuations across the entire parameter range. While classical and semi-classical frameworks predict noncoplanar orders competing with a collinear Néel state, we find these phases to be fragile: once full quantum fluctuations are included, they melt into a quantum-entangled state characterized by suppressed spin correlations and enhanced entanglement entropy. Our findings highlight how competition between qualitatively different quantum disordered phases provides a fertile playground for unconventional phases emerging from their interplay and quantum fluctuations.

Figures (7)

• Figure 1: (a) Schematic illustration of the $S=3/2$ Kitaev QSL on the honeycomb lattice in the SO(6) Majorana representation. Each $S=3/2$ spin (large gray circle) is decomposed into three itinerant Majorana fermions (small circles) and three gauge Majorana fermions (triangles). The gauge Majorana fermions define $\mathbb{Z}_2$ conserved quantities on each bond, indicated by red, green, and blue bonds. (b) Graphical representation of the AKLT VBS state on the honeycomb lattice. Each $S=3/2$ spin (large gray circle) is decomposed into three $S=1/2$ spins (small orange circles), and each pair of neighboring $S=1/2$ spins forms a singlet state (blue line). Projecting the three $S=1/2$ spins at each site onto the fully symmetric subspace yields the AKLT VBS state. (c) Ground-state phase diagrams of the Kitaev-AKLT honeycomb model defined in Eq. \ref{['eq:H']} obtained by three complementary methods: classical O(3) vector analysis (inner circle), semi-classical SU(4) coherent state approach (middle circle), and full quantum ED (outer circle). Each circle displays the second derivative of the ground state energy $E_0$ with respect to $\xi$, $-\frac{\partial^2 E_0}{\partial \xi^2}$. For ED, the solid and dashed lines correspond to the results for $N=12$ and $N=8$ clusters [see also the inset of Fig. \ref{['fig:EE']}], respectively; the $N=8$ data are multiplied by a factor of 10 for visibility. See the text for details.
• Figure 2: $\xi$ dependencies of (a) the normalized spin structure factor $S(\mathbf{q})/N$ at the $\Gamma$, M, X, and $\Gamma^\prime$ points and (b) the entanglement entropy (EE), obtained from ED for $N=8$ and $N=12$ clusters. The inset of (a) represents the Brillouin zone of the honeycomb lattice, with the inner and outer hexagons indicating the first Brillouin zone and the extended one up to the third zone, respectively. The inset of (b) illustrates the cluster shapes and bipartition scheme used for the EE calculations.
• Figure 3: Momentum space distributions of $S(\mathbf{q})/N$ up to the third Brillouin zone for $\xi=0.75$, $0.80$, $0.90$, and $1.00$ on $N=8$ (upper panels) and $N=12$ (lower panels) clusters, obtained from the ED calculations. Inner white hexagons indicate the first Brillouin zone.
• Figure S1: $\xi$ dependencies of the normalized spin structure factor, $S(\mathbf{q})/N$, at the $\Gamma$, M, X, and $\Gamma^\prime$ points, obtained by (a) classical and (b) semi-classical approaches. The inset of (a) represents the Brillouin zone of the honeycomb lattice, with the inner and outer hexagons indicating the first Brillouin zone and the extended zone up to the third one, respectively.
• Figure S2: Comparison of (a) cant-Néel, (b) 3$Q$ chiral, and (c) 2$Q$ orders. The upper panels show the momentum space distributions of $S(\mathbf{q})/N$, where the Bragg peaks are represented by circles for better visibility. The lower panels display the corresponding real-space spin configurations along with the values of the scalar spin chirality $\kappa$.