Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OD-Stega: LLM-Based Relatively Secure Steganography via Optimized Distributions

Yu-Shin Huang, Peter Just, Hanyun Yin, Krishna Narayanan, Ruihong Huang, Chao Tian

TL;DR

OD-Stega presents a principled framework for relatively secure, coverless steganography using LLMs by maximizing the entropy of the next-token distribution $H(Q^{i})$ under a divergence constraint $D(Q^{i} || P^{i}) \le \delta$, with closed-form solutions for $D_{KL}$ and $D_{TV}$ and a demonstrated equivalence between KL-based optimization and temperature scaling on logits. The method includes practical enhancements for tokenization mismatch, vocabulary truncation with additive KL properties, and compatibility with non-AC approaches like Discop, yielding flexible, high-capacity steganography while controlling detectability through a tunable budget $\delta$ and adaptive $\delta_i = C\,H(P^{i})$. Experimental results using LLAMA2-7B and GPT2-XL show meaningful embedding gains (e.g., ~20%–55% increases in bits embedded per 25 tokens) and provide a multi-faceted evaluation of imperceptibility via steganalysis and GPT-4 human-like judgments, with longer sequences generally more detectable. OD-Stega thus offers a scalable, tunable framework for relatively secure covert communication in natural language, adaptable to existing protocols (e.g., OD-Discop) and practical deployment constraints.

Abstract

We consider coverless steganography where a Large Language Model (LLM) is used to generate stego-texts in combination with arithmetic coding. An efficient method should embed secret bits in as few language tokens as possible while keeping the stego-text as natural as possible. We show that this problem is equivalent to maximizing the entropy of a replacement probability distribution of the next token generation, subject to a constraint on the divergence between the new distribution and the original one produced by the LLM. A closed-form solution is provided under either the KL divergence or the total variation constraint. Several important practical issues are also tackled: 1) An often-overlooked tokenization mismatch issue is resolved with a simple prompt selection approach, 2) The combination of the optimized distribution and the vocabulary truncation technique is considered, and 3) The incorporation of the proposed approach with existing (potentially non arithmetic coding based) techniques, e.g., the Discop technique.

OD-Stega: LLM-Based Relatively Secure Steganography via Optimized Distributions

TL;DR

OD-Stega presents a principled framework for relatively secure, coverless steganography using LLMs by maximizing the entropy of the next-token distribution under a divergence constraint , with closed-form solutions for and and a demonstrated equivalence between KL-based optimization and temperature scaling on logits. The method includes practical enhancements for tokenization mismatch, vocabulary truncation with additive KL properties, and compatibility with non-AC approaches like Discop, yielding flexible, high-capacity steganography while controlling detectability through a tunable budget and adaptive . Experimental results using LLAMA2-7B and GPT2-XL show meaningful embedding gains (e.g., ~20%–55% increases in bits embedded per 25 tokens) and provide a multi-faceted evaluation of imperceptibility via steganalysis and GPT-4 human-like judgments, with longer sequences generally more detectable. OD-Stega thus offers a scalable, tunable framework for relatively secure covert communication in natural language, adaptable to existing protocols (e.g., OD-Discop) and practical deployment constraints.

Abstract

We consider coverless steganography where a Large Language Model (LLM) is used to generate stego-texts in combination with arithmetic coding. An efficient method should embed secret bits in as few language tokens as possible while keeping the stego-text as natural as possible. We show that this problem is equivalent to maximizing the entropy of a replacement probability distribution of the next token generation, subject to a constraint on the divergence between the new distribution and the original one produced by the LLM. A closed-form solution is provided under either the KL divergence or the total variation constraint. Several important practical issues are also tackled: 1) An often-overlooked tokenization mismatch issue is resolved with a simple prompt selection approach, 2) The combination of the optimized distribution and the vocabulary truncation technique is considered, and 3) The incorporation of the proposed approach with existing (potentially non arithmetic coding based) techniques, e.g., the Discop technique.
Paper Structure (38 sections, 4 theorems, 23 equations, 21 figures, 8 tables)

This paper contains 38 sections, 4 theorems, 23 equations, 21 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. LLM-based Steganography
  4. Arithmetic Coding
  5. Proposed Methodology
  6. Optimized Distribution under Constraint
  7. Optimal Probability Adjustment
  8. Adaptive $\delta$ Value Selection
  9. Practical Considerations and Variations
  10. Tokenization Errors
  11. Vocabulary Truncation
  12. Variations and OD-Discop
  13. Experimental Results
  14. Experiment Setup
  15. Utilization-KL Tradeoff
  16. ...and 23 more sections

Key Result

Theorem 1

When the divergence in (eq:KLconstraint) is the KL divergence, the optimal solution $Q^{i}$ to the problem (eq:HPX)-(eq:px_equal_0) is obtained by performing temperature scaling on $P^{i}$ when $\delta \in [0,\frac{1}{N_i} \sum_{j =1}^{N_i} \log(\frac{1}{N_i P_j^{i}}) ]$ for some $T \geq 1$ s.t. $D_{KL}(Q^{i}||P^{i}) = \delta$. Otherwise,

Figures (21)

  • Figure 1: Example of AC in steganography: The sequence $10111$ can be represented as the interval $\mathbf{I} = [0.101110000 \cdots_{2}, 0.1011111111\cdots_{2}) \simeq [0.71875, 0.75)$. We identify the range where this interval falls in the probability distribution $P^{i}$.
  • Figure 2: Optimal adjustment allocation under total variation constraint.
  • Figure 3: The OD-Stega approach
  • Figure 4: The two-stage design: Vocabulary truncation and distribution optimization
  • Figure 5: OD-Stega. Average bits embedded per 25 tokens (over 200 stego-texts) under KL and TV constraints, across parameter $C$ and cutoff $\epsilon$.
  • ...and 16 more figures

Theorems & Definitions (5)

  • Theorem 1
  • Remark 1
  • Lemma 1
  • Theorem 2: informal
  • Theorem 3