HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DST

TL;DR

HiCoLoRA tackles semantic misalignment in zero-shot dialog state tracking by introducing a hierarchical collaborative Low-Rank Adaptation framework that separates domain-agnostic and domain-specific knowledge. It combines UniRep-LoRA for universal context, SemAdapt-LoRA for domain prompts, an Adaptive Linear Fusion to balance them, spectral clustering for domain-slot disentanglement, and SemSVD-Init to preserve pre-trained knowledge, together forming a robust context-prompt alignment mechanism. The approach yields state-of-the-art performance on MultiWOZ and SGD in zs-DST, with strong cross-domain generalization and efficiency comparable to standard LoRA. This work offers a scalable, parameter-efficient strategy for zero-shot TOD deployment, reducing data requirements while maintaining high slot-inference fidelity across diverse domains; future work may further improve rare-slot handling and boundary detection.

Abstract

Zero-shot Dialog State Tracking (zs-DST) is essential for enabling Task-Oriented Dialog Systems (TODs) to generalize to new domains without costly data annotation. A central challenge lies in the semantic misalignment between dynamic dialog contexts and static prompts, leading to inflexible cross-layer coordination, domain interference, and catastrophic forgetting. To tackle this, we propose Hierarchical Collaborative Low-Rank Adaptation (HiCoLoRA), a framework that enhances zero-shot slot inference through robust prompt alignment. It features a hierarchical LoRA architecture for dynamic layer-specific processing (combining lower-layer heuristic grouping and higher-layer full interaction), integrates Spectral Joint Domain-Slot Clustering to identify transferable associations (feeding an Adaptive Linear Fusion Mechanism), and employs Semantic-Enhanced SVD Initialization (SemSVD-Init) to preserve pre-trained knowledge. Experiments on multi-domain datasets MultiWOZ and SGD show that HiCoLoRA outperforms baselines, achieving SOTA in zs-DST. Code is available at https://github.com/carsonz/HiCoLoRA.

Figures (9)

• Figure 1: Three critical challenges motivating our work: (1) Architectural rigidity hinders cross-layer coordination in Transformers, limiting fine-grained semantic alignment; (2) Coupling of domain-shared and domain-specific semantics causes cross-domain confusion; (3) Random parameter initialization distorts pre-trained knowledge, exacerbating catastrophic forgetting.
• Figure 2: The HiCoLoRA framework combines: (1) UniRep-LoRA and SemAdapt-LoRA with Adaptive Linear Fusion balancing domain-agnostic and domain-specific features; (2) Spectral Joint Domain-Slot Clustering disentangling domain semantics to guide fusion; (3) SemSVD-Init preserving pre-trained knowledge via singular value modulation. These synergistically address context-prompt misalignment, enhancing zero-shot slot inference.
• Figure 3: Accuracy of HiCoLoRA with different rank on the MultiWOZ dataset.
• Figure 4: Accuracy of different high layer ratio (full collaboration) in HiCoLoRA.
• Figure 5: Example Attention Maps of the First and Last Transformer Layers in HiCoLoRA.