A Generative Security Application Engineering Curriculum

TL;DR

This work addresses the need to align security education with rapid advances in generative AI and LLMs by proposing an initial curriculum that teaches students to build and secure LLM-enabled security applications. It frames a four-part course around model exploration, practical prototyping with frameworks (notably LangChain), security considerations, and task-based labs with ground-truth results to mitigate model fallibility. Key contributions include a detailed curricular structure (models, frameworks, RAG, agents, alternative frameworks, security, and tasks), hands-on modules for code analysis, vulnerability discovery, and threat intelligence, and explicit attention to secure tool design and prompt-resilience. The curriculum, tested at Portland State University with publicly available teaching materials, aims to provide a scalable blueprint for institutions to prepare students for a generative AI–driven security landscape and to foster reproducible, security-conscious AI education with practical, real-world tasks.

Abstract

Generative AI and large language models (LLMs) are transforming security by automating many tasks being performed manually. With such automation changing the practice of security as we know it, it is imperative that we prepare future students for the technology landscape they will ultimately face. Towards this end, we describe an initial curriculum and course that attempts to show students how to apply generative AI in order to solve problems in security. By refocusing security education and training on aspects uniquely suited for humans and showing students how to leverage automation for the rest, we believe we can better align security education practices with generative AI as it evolves.

Figures (6)

• Figure 1: Course overview
• Figure 2: Vulnerable SQL agent
• Figure 3: Securing LLM agents with custom tools
• Figure 4: ReAct prompt template injection locations
• Figure 5: Prompt injection of ReAct template