Eliciting Least-to-Most Reasoning for Phishing URL Detection

TL;DR

This work addresses phishing URL detection by leveraging an interpretable, multi-step reasoning approach based on Least-to-Most prompting. It introduces an answer sensitivity mechanism that provides iterative phishing likelihood estimates, guiding sub-question resolution within a cap of 10 iterations. Across three URL datasets and four LLMs, the method consistently outperforms a one-shot baseline and approaches the performance of a supervised model (URLTran) while requiring minimal labeled data. The results demonstrate that structured prompt-driven reasoning can closely bridge the gap to supervised detectors and offer clearer insights into the model’s decision process.

Abstract

Phishing continues to be one of the most prevalent attack vectors, making accurate classification of phishing URLs essential. Recently, large language models (LLMs) have demonstrated promising results in phishing URL detection. However, their reasoning capabilities that enabled such performance remain underexplored. To this end, in this paper, we propose a Least-to-Most prompting framework for phishing URL detection. In particular, we introduce an "answer sensitivity" mechanism that guides Least-to-Most's iterative approach to enhance reasoning and yield higher prediction accuracy. We evaluate our framework using three URL datasets and four state-of-the-art LLMs, comparing against a one-shot approach and a supervised model. We demonstrate that our framework outperforms the one-shot baseline while achieving performance comparable to that of the supervised model, despite requiring significantly less training data. Furthermore, our in-depth analysis highlights how the iterative reasoning enabled by Least-to-Most, and reinforced by our answer sensitivity mechanism, drives these performance gains. Overall, we show that this simple yet powerful prompting strategy consistently outperforms both one-shot and supervised approaches, despite requiring minimal training or few-shot guidance. Our experimental setup can be found in our Github repository github.sydney.edu.au/htri0928/least-to-most-phishing-detection.

Figures (4)

• Figure 1: General Least-to-Most prompting framework.
• Figure 2: Prompt asking the model to break down into sub-questions of how to answer whether the given URL is phishing or benign based on the described answer sensitivity method.
• Figure 3: Distribution of number of iterations (GPT-EBBU)
• Figure 4: Trajectory of sensitivity values (GPT-EBBU).