OSINT Clinic: Co-designing AI-Augmented Collaborative OSINT Investigations for Vulnerability Assessment

TL;DR

OSINT Clinic addresses the scalability gap in vulnerability assessments for small businesses by leveraging publicly available data and AI-augmented collaboration. The authors adapt the Matchmaking for AI co-design method into a longitudinal, three-study design with six undergraduates and three real client engagements to identify challenges and validate AI-enabled workflows. Key contributions include a conceptual OSINT clinic program, methodological extensions for collaboration and learning, empirical insights on learning goals and team dynamics, and design considerations addressing privacy, workflow integration, and leadership in AI-assisted OSINT. The pilot demonstrates AI can streamline OSINT investigations and produce actionable client outputs, while also highlighting privacy concerns and monitoring challenges that warrant careful design and governance.

Abstract

Small businesses need vulnerability assessments to identify and mitigate cyber risks. Cybersecurity clinics provide a solution by offering students hands-on experience while delivering free vulnerability assessments to local organizations. To scale this model, we propose an Open Source Intelligence (OSINT) clinic where students conduct assessments using only publicly available data. We enhance the quality of investigations in the OSINT clinic by addressing the technical and collaborative challenges. Over the duration of the 2023-24 academic year, we conducted a three-phase co-design study with six students. Our study identified key challenges in the OSINT investigations and explored how generative AI could address these performance gaps. We developed design ideas for effective AI integration based on the use of AI probes and collaboration platform features. A pilot with three small businesses highlighted both the practical benefits of AI in streamlining investigations, and limitations, including privacy concerns and difficulty in monitoring progress.

Figures (5)

• Figure 1: Study Diagram: (1) Study 1 was formative and aimed at identifying the key challenges in the Intelligence Cycle phases of OSINT investigations. We asked participants to reflect on previous investigations and document the steps and challenges faced during Design Workshop 1. (2) In Study 2, we introduced generative AI probes in Design Workshops 2--4 to explore how AI could address challenges experienced earlier. During the workshops, we identified areas where generative AI was effective and also its limitations. We focused on the emergent challenges of collaboration and leadership during Design Workshops 5 and 6, with a collaborative AI platform (Team-GPT) as a probe. (3) In Study 3, we piloted OSINT-based vulnerability assessments with three small businesses, integrating curated generative AI prompts into ongoing investigations. This real-world application highlighted the practical utility and surfaced challenges of using foundational generative AI models like ChatGPT in vulnerability assessments, especially in collaborative settings.
• Figure 2: OSINT Intelligence Model in the context of cybersecurity as described by Hwang et al. hwang2022current
• Figure 3: Team-GPT platform team-gpt_enterprise_nodate with core features: (a) personal chats for individual work and shared workspaces for team collaboration. Chats were grouped by the different phases of OSINT investigation in our case; (b) a team library to store shared prompt templates which can be invoked with a click after filling in the placeholders. We used the library to curate a set of useful prompts for the participants in Study 3; (c) directly comment on prompts and responses to provide feedback; (d) fork chats to new personal or shared chats to continue the conversation.
• Figure 4: FigJam board showing the canvas for brainstorming during Design Workshop 1
• Figure 5: FigJam board showing the canvas and results for Learning Goal 2 during Design Workshop 2