SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfaces

TL;DR

This work tackles privacy and safety in LLM-based NLIDB by addressing stealthy, multi-turn leakage risks that existing defenses fail to mitigate without sacrificing SQL utility. It introduces SafeNlidb, an end-to-end privacy-security alignment framework with two core components: Security-Aware NLIDB Data Synthesis, which automatically generates secure interaction data and yields the ShieldSQL benchmark, and Alternating Preference Optimization (APO) with a Reasoning Warm-Up that stabilizes multi-objective learning without manual labels. A Hybrid Chain-of-Thought (H-CoT) mechanism combines safety reasoning and SQL generation, and APO empirically balances security and utility, outperforming larger models and decoupled-expert pipelines on both security and reliability metrics. The approach enables private deployment, maintains high SQL execution accuracy, and reveals valuable insights through ablations, robustness analyses, and comprehensive comparisons to DP and rule-based defenses across multi-turn NLIDB scenarios. These advances offer a practical path toward secure, private NLIDB deployments in real-world database systems.

Abstract

The rapid advancement of Large Language Models (LLMs) has driven significant progress in Natural Language Interface to Database (NLIDB). However, the widespread adoption of LLMs has raised critical privacy and security concerns. During interactions, LLMs may unintentionally expose confidential database contents or be manipulated by attackers to exfiltrate data through seemingly benign queries. While current efforts typically rely on rule-based heuristics or LLM agents to mitigate this leakage risk, these methods still struggle with complex inference-based attacks, suffer from high false positive rates, and often compromise the reliability of SQL queries. To address these challenges, we propose \textsc{SafeNlidb}, a novel privacy-security alignment framework for LLM-based NLIDB. The framework features an automated pipeline that generates hybrid chain-of-thought interaction data from scratch, seamlessly combining implicit security reasoning with SQL generation. Additionally, we introduce reasoning warm-up and alternating preference optimization to overcome the multi-preference oscillations of Direct Preference Optimization (DPO), enabling LLMs to produce security-aware SQL through fine-grained reasoning without the need for human-annotated preference data. Extensive experiments demonstrate that our method outperforms both larger-scale LLMs and ideal-setting baselines, achieving significant security improvements while preserving high utility. WARNING: This work may contain content that is offensive and harmful!

Figures (15)

• Figure 1: (a) Examples of sensitive data leakage in NLIDB: easily detectable Direct Attacks and stealthy Inference-Based Attacks (Complementary Queries). (b) Comparison of SafeNlidb with previous LLM-based NLIDB methods.
• Figure 2: The overall framework of SafeNlidb. (a) Security-aware NLIDB interaction data synthesis process. (b) Reasoning warm-up and alternating preference optimization performed on the synthesized data from (a).
• Figure 3: Evaluating the impact of interaction rounds on model security and reliability in the ShieldSQL dataset.
• Figure 4: Comparison between SafeNlidb (Llama3-8B) and various Decoupled Experts on the SecureSQL dataset. The form of A + B denotes that A performs safety assessment while B handles Text-to-SQL generation. E(8B/70B) represents the vanilla Llama3-8B/70B, E*(8B) refers to Llama3-8B trained with our synthesized Safety-CoT or SQL-CoT.
• Figure 5: Distribution of different interaction scenario types in the test set of ShieldSQL dataset.