From Reviewers' Lens: Understanding Bug Bounty Report Invalid Reasons with LLMs

TL;DR

The paper addresses the challenge of validating bug bounty reports amid rising AI-generated submissions by evaluating large language models and introducing a reviewer-knowledge augmentation approach. It demonstrates that while LLMs achieve high overall accuracy, they biasedly favor valid reports and miss many invalid ones; incorporating a taxonomy of invalid reasons and retrieval-augmented generation (RAG) with vendor scope improves detection consistency, particularly for Information Disclosure cases. The study also reveals that reviewer reputation can subtly influence outcomes in borderline cases, highlighting human factors in review decisions. Together, the findings suggest that combining LLM capabilities with structured reviewer knowledge can yield more transparent, fair, and efficient vulnerability report reviews with practical implications for bug bounty platforms.

Abstract

Bug bounty platforms (e.g., HackerOne, BugCrowd) leverage crowd-sourced vulnerability discovery to improve continuous coverage, reduce the cost of discovery, and serve as an integral complement to internal red teams. With the rise of AI-generated bug reports, little work exists to help bug hunters understand why these reports are labeled as invalid. To improve report quality and reduce reviewers' burden, it is critical to predict invalid reports and interpret invalid reasons. In this work, we conduct an empirical study with the purpose of helping bug hunters understand the validity of reports. We collect a dataset of 9,942 disclosed bug bounty reports, including 1,400 invalid reports, and evaluate whether state-of-the-art large language models can identify invalid reports. While models such as GPT-5, DeepSeek, and a fine-tuned RoBERTa achieve strong overall accuracy, they consistently struggle to detect invalid cases, showing a tendency to over-accept reports. To improve invalidity detection, we build a taxonomy of rejection reasons for Information Disclosure vulnerabilities and incorporate it into a retrieval-augmented generation (RAG) framework. This approach substantially improves classification consistency and reduces bias. We also examine whether reviewer decisions may be influenced by factors beyond the content of the report. Our analysis shows that reporters with higher reputations tend to receive more favorable outcomes in borderline cases, suggesting that perceived expertise can influence review judgments. Overall, our findings highlight the challenges of invalid report identification and show that combining LLMs with structured reviewer knowledge can support more transparent and consistent vulnerability report review.

Figures (4)

• Figure 1: Example invalid bug bounty report (#460642, HackerOne hackerone_460642). The submission, titled "HTTP PUT method enabled," was closed as Spam because it reused a previously disclosed demo.
• Figure 2: Example of vendor scope entries on HackerOne (Palo Alto Software program) hackerone_paloalto_policy_scopes. Each entry defines the in-scope and out-of-scope assets for valid submissions.
• Figure 3: Aligned: the higher-reputation reporter received a higher-ranked status (e.g., Resolved over Informative); Equal: both had identical reputation; Reversed: the higher-reputation reporter received a lower-ranked status.
• Figure 4: Decision boundary comparison between high and low reputation reporters.