Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation

TL;DR

This work tackles zero-shot cross-domain dialogue state tracking by introducing DualLoRA, a dual Low-Rank Adaptation framework that separately tunes the dialogue context and slot prompts. By integrating two LoRA modules into the attention mechanism and merging prompt updates into model biases, DualLoRA preserves prompt influence across transformer layers without adding inference latency, enabling efficient cross-domain transfer. Evaluations on MultiWOZ and SGD show consistent improvements in Joint Goal Accuracy ($JGA$) and Average Goal Accuracy ($AGA$) over strong baselines, with ablations confirming the contributions of both ContextLoRA and PromptLoRA. The approach offers a practical, plug-and-play solution for enhancing zero-shot DST in task-oriented dialogue systems, with code made available for reuse.

Abstract

Zero-shot dialogue state tracking (DST) seeks to enable dialogue systems to transition to unfamiliar domains without manual annotation or extensive retraining. Prior research has approached this objective by embedding prompts into language models (LMs). Common methodologies include integrating prompts at the input layer or introducing learnable variables at each transformer layer. Nonetheless, each strategy exhibits inherent limitations. Prompts integrated at the input layer risk underutilization, with their impact potentially diminishing across successive transformer layers. Conversely, the addition of learnable variables to each layer can complicate the training process and increase inference latency. To tackle the issues mentioned above, this paper proposes Dual Low-Rank Adaptation (DualLoRA), a plug-and-play architecture designed for zero-shot DST. DualLoRA incorporates two distinct Low-Rank Adaptation (LoRA) components, targeting both dialogue context processing and prompt optimization, to ensure the comprehensive influence of prompts throughout the transformer model layers. This is achieved without incurring additional inference latency, showcasing an efficient integration into existing architectures. Through rigorous evaluation on the MultiWOZ and SGD datasets, DualLoRA demonstrates notable improvements across multiple domains, outperforming traditional baseline methods in zero-shot settings. Our code is accessible at: \url{https://github.com/suntea233/DualLoRA}.

Figures (26)

• Figure 1: Example of a multi-domain dialogue from MultiWOZ dataset.
• Figure 2: Different Architectures for Zero-Shot Dialogue State Tracking.
• Figure 3: T5DST lin_leveraging_2021 attention map between dialogue context and prompts in the first layer and last layer of the encoder. The last encoder layer allocates less attention to prompt tokens.
• Figure 4: The overall architecture of DualLoRA.
• Figure 5: Performance of DualLoRA with different fusion methods on the MultiWOZ dataset.