LLMAID: Identifying AI Capabilities in Android Apps with LLMs
Pei Liu, Terry Zhuo, Jiawei Deng, Thong James, Shidong Pan, Sherry Xu, Zhenchang Xing, Qinghua Lu, Xiaoning Du, Hongyu Zhang
TL;DR
LLMAID automates AI Discovery in Android apps by combining static analysis with LLM-based interpretation to identify AI components and generate user-facing summaries. It introduces four tasks—candidate extraction, knowledge base interaction, AI capability analysis and detection, and AI service summarization—and validates them on large-scale datasets, achieving precision and recall above 90% and identifying AI apps at least twice as many as a state-of-the-art rule-based baseline. A user study shows developers prefer LLMAID's AI summaries for informativeness and usefulness. The study reveals a concentration of AI capabilities in computer vision and data processing, offering actionable insights for researchers and practitioners on AI deployment in mobile apps.
Abstract
Recent advancements in artificial intelligence (AI) and its widespread integration into mobile software applications have received significant attention, highlighting the growing prominence of AI capabilities in modern software systems. However, the inherent hallucination and reliability issues of AI continue to raise persistent concerns. Consequently, application users and regulators increasingly ask critical questions such as: Does the application incorporate AI capabilities? and What specific types of AI functionalities are embedded? Preliminary efforts have been made to identify AI capabilities in mobile software; however, existing approaches mainly rely on manual inspection and rule-based heuristics. These methods are not only costly and time-consuming but also struggle to adapt advanced AI techniques. To address the limitations of existing methods, we propose LLMAID (Large Language Model for AI Discovery). LLMAID includes four main tasks: (1) candidate extraction, (2) knowledge base interaction, (3) AI capability analysis and detection, and (4) AI service summarization. We apply LLMAID to a dataset of 4,201 Android applications and demonstrate that it identifies 242% more real-world AI apps than state-of-the-art rule-based approaches. Our experiments show that LLM4AID achieves high precision and recall, both exceeding 90%, in detecting AI-related components. Additionally, a user study indicates that developers find the AI service summaries generated by LLMAID to be more informative and preferable to the original app descriptions. Finally, we leverage LLMAID to perform an empirical analysis of AI capabilities across Android apps. The results reveal a strong concentration of AI functionality in computer vision (54.80%), with object detection emerging as the most common task (25.19%).