Nevermind: Instruction Override and Moderation in Large Language Models

TL;DR

The paper investigates instruction following in conflicting scenarios across a spectrum of LLMs, focusing on overrides that target internal model weights, prompt-derived context, and jailbreak prompts. It extends a needle-in-a-haystack framework to three override modalities and analyzes performance as model size and context length scale, incorporating rope scaling to extend context up to 12k tokens. Key findings show larger models (e.g., GPT-4 and Tess XL 120B) exhibit stronger instruction-following but are more susceptible to jailbreak prompts, while context-length expansion demands a deliberate perplexity-buffer to avoid degradation in retrieval. The work argues that aligning safety with instruction following may require external safeguards beyond the LLM, proposing a neuro-inspired external-control framework to enhance safe and trustworthy AI deployment. These insights have practical implications for designing robust guardrails and evaluating safety in large-scale language models.

Abstract

Given the impressive capabilities of recent Large Language Models (LLMs), we investigate and benchmark the most popular proprietary and different sized open source models on the task of explicit instruction following in conflicting situations, e.g. overrides. These include the ability of the model to override the knowledge within the weights of the model, the ability to override (or moderate) extracted knowledge in the prompt, and lastly the ability to perform a full jailbreak. Experimentation performed suggest several key findings to improve instruction following - larger models perform the best in following instructions that override internal and contextual instructions, and are obedient, even to a fault. When scaling to longer contexts via rope scaling, a significant buffer needs to be maintained from the edge of the perplexity cliff in order to maintain instruction following capabilities. Finally, we observe improving instruction following, and subsequently instruction overrides/jailbreaks, is fundamentally at odds with the ability of a language model to follow given safety filters or guidelines. Thus, we postulate the most effective approach for safe, trustworthy AI should be dealt external to the LLM itself.

Figures (5)

• Figure 1: Visualization of the ability of different LLMs to find a "needle in a haystack", original work by user "gkamradt" needle. We extend this evaluation to instruction overrides, and evaluate different parameter sized LLMs on the ability to override knowledge "baked" into the weights of the model, override extracted knowledge from the context, and also perform a full jailbreak of previous instruction. The best performance was demonstrated from the 120B parameter Tess XL model tessxl.
• Figure 2: Plot of the perplexity of models increasing in parameter size and context sizes. We test 7B, 13B, 70B, and 120B parameter models and plot the perplexity of an average of 10 long context strings from the wikitext dataset. By tracking the perplexity cliff (exploding perplexity inflection point), we can find the optimal $\alpha$ and linear regressed 2nd order polynomial that fits the given datapoints.
• Figure 3: Illustrations of the needle in a haystack test using 8k and 12k rope scaled context windows using a static and dynamic rope base change. The best performance is at $\beta$=1.5, where the perplexity cliff is pushed a safe distance away from the retrieved information in the context window.
• Figure 4: Conversation with token based moderation. Simply restricting a set of words is insufficient for content moderation as the LLM can adeptly substitute similar words or misspellings. Substitution with redirection by inserting, "... nevermind, let" is better, but it still maintains remnants of the concept.
• Figure 5: Moderation by inhibition of the LLM thoughts via a main LLM and speculative decoder LLM. If the LLM thinks about a certain topic that should be filtered, it inhibits all of those tokens in the sampler of the main output. In this way, the LLM "thinks before it speaks".