Safety Training Persists Through Helpfulness Optimization in LLM Agents

TL;DR

The effects of running direct preference optimization on safety or helpfulness alone vs both metrics sequentially vs both metrics sequentially is compared and the need for better understanding of post-training dynamics is underscored.

Abstract

Safety post-training has been studied extensively in single-step "chat" settings where safety typically refers to refusing harmful requests. We study an "agentic" (i.e., multi-step, tool-use) setting where safety refers to harmful actions directly taken by the LLM. We compare the effects of running direct preference optimization (DPO) on safety or helpfulness alone vs both metrics sequentially. As expected, training on one metric alone results in an extreme point along this frontier. However, unlike prior work, we find that safety training persists through subsequent helpfulness training. We also find that all training configurations end up near a linear Pareto frontier with $R^2 = 0.77$. Even post-training on both metrics simultaneously simply results in another point on the frontier rather than finding a "best of both worlds" strategy, despite the presence of such strategies in our DPO dataset. Overall, our findings underscore the need for better understanding of post-training dynamics.

Figures (16)

• Figure 1: The ToolEmu execution flow for a single task. First, the instruction and the specifications of the available tools are given to the agent LLM. Then on each time step, the agent selects an action (an available tool and a tool input). The "emulator LLM" simulates the outcome of the action and provides the resulting observation to the agent. Once the agent declares that the task is complete by using the phrase "Final Answer", the transcript is saved as a trajectory. Lastly, an "evaluator LLM" assigns safety and helpfulness scores. Unlike the agent, the evaluator has access to an explicit pre-written list of underspecifications and potential risks which is not given to the agent.
• Figure 2: An illustration of our experiment pipeline. \ref{['sec:setup']} provides a detailed explanation.
• Figure 3: An illustration of safety persistence (left) and helpfulness persistence (right).
• Figure 4: Safety and helpfulness scores for all training runs.
• Figure 5: Safety and helpfulness deltas between each post-trained model and its associated source model.