BlackIce: A Containerized Red Teaming Toolkit for AI Security Testing

TL;DR

AI red-teaming tooling is fragmented, with conflicting dependencies and varied runtimes that complicate reproducible assessments. BlackIce announces a containerized, open-sourceDocker image that bundles 14 AI security tools into a unified, version-pinned environment with static and dynamic tool categories, automated installation, patching, and community extensibility. It aligns tool coverage with MITRE ATLAS and the Databricks AI Security Framework (DASF), providing structured evaluation capabilities and a pathway for easy extension. The solution reduces setup effort, enables cloud/notebook usage, and promotes standardized, responsible AI security testing across organizations.

Abstract

AI models are being increasingly integrated into real-world systems, raising significant concerns about their safety and security. Consequently, AI red teaming has become essential for organizations to proactively identify and address vulnerabilities before they can be exploited by adversaries. While numerous AI red teaming tools currently exist, practitioners face challenges in selecting the most appropriate tools from a rapidly expanding landscape, as well as managing complex and frequently conflicting software dependencies across isolated projects. Given these challenges and the relatively small number of organizations with dedicated AI red teams, there is a strong need to lower barriers to entry and establish a standardized environment that simplifies the setup and execution of comprehensive AI model assessments. Inspired by Kali Linux's role in traditional penetration testing, we introduce BlackIce, an open-source containerized toolkit designed for red teaming Large Language Models (LLMs) and classical machine learning (ML) models. BlackIce provides a reproducible, version-pinned Docker image that bundles 14 carefully selected open-source tools for Responsible AI and Security testing, all accessible via a unified command-line interface. With this setup, initiating red team assessments is as straightforward as launching a container, either locally or using a cloud platform. Additionally, the image's modular architecture facilitates community-driven extensions, allowing users to easily adapt or expand the toolkit as new threats emerge. In this paper, we describe the architecture of the container image, the process used for selecting tools, and the types of evaluations they support.

Figures (1)

• Figure 1: Flowchart illustrating the Docker build process for the BlackIce container image. Boxes with red borders highlight the steps where users directly interact with the build process to integrate new tools.