Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating LLM-Based Mobile App Recommendations: An Empirical Study

Quim Motger, Xavier Franch, Vincenzo Gervasi, Jordi Marco

TL;DR

The paper investigates how commercial LLMs generate, justify, and rank mobile app recommendations via prompt-based queries, focusing on explainability, reproducibility, and alignment with ASO signals. By eliciting ranking criteria across multiple models and queries, the authors derive a taxonomy of 16 criteria and develop a framework to assess internal and external consistency, as well as the impact of explicit criteria on outputs. Key findings show LLMs rely on a broad, often subjective set of criteria with only limited overlap with traditional ASO metrics, and that explicit prompts can steer recommendations, though adherence is inconsistent and model-dependent. The study provides a replication package to support reproducibility and offers practical guidance for end-users, developers, and researchers seeking transparent, controllable, and diverse AI-assisted app discovery.

Abstract

Large Language Models (LLMs) are increasingly used to recommend mobile applications through natural language prompts, offering a flexible alternative to keyword-based app store search. Yet, the reasoning behind these recommendations remains opaque, raising questions about their consistency, explainability, and alignment with traditional App Store Optimization (ASO) metrics. In this paper, we present an empirical analysis of how widely-used general purpose LLMs generate, justify, and rank mobile app recommendations. Our contributions are: (i) a taxonomy of 16 generalizable ranking criteria elicited from LLM outputs; (ii) a systematic evaluation framework to analyse recommendation consistency and responsiveness to explicit ranking instructions; and (iii) a replication package to support reproducibility and future research on AI-based recommendation systems. Our findings reveal that LLMs rely on a broad yet fragmented set of ranking criteria, only partially aligned with standard ASO metrics. While top-ranked apps tend to be consistent across runs, variability increases with ranking depth and search specificity. LLMs exhibit varying sensitivity to explicit ranking instructions - ranging from substantial adaptations to near-identical outputs - highlighting their complex reasoning dynamics in conversational app discovery. Our results aim to support end-users, app developers, and recommender-systems researchers in navigating the emerging landscape of conversational app discovery.

Evaluating LLM-Based Mobile App Recommendations: An Empirical Study

TL;DR

The paper investigates how commercial LLMs generate, justify, and rank mobile app recommendations via prompt-based queries, focusing on explainability, reproducibility, and alignment with ASO signals. By eliciting ranking criteria across multiple models and queries, the authors derive a taxonomy of 16 criteria and develop a framework to assess internal and external consistency, as well as the impact of explicit criteria on outputs. Key findings show LLMs rely on a broad, often subjective set of criteria with only limited overlap with traditional ASO metrics, and that explicit prompts can steer recommendations, though adherence is inconsistent and model-dependent. The study provides a replication package to support reproducibility and offers practical guidance for end-users, developers, and researchers seeking transparent, controllable, and diverse AI-assisted app discovery.

Abstract

Large Language Models (LLMs) are increasingly used to recommend mobile applications through natural language prompts, offering a flexible alternative to keyword-based app store search. Yet, the reasoning behind these recommendations remains opaque, raising questions about their consistency, explainability, and alignment with traditional App Store Optimization (ASO) metrics. In this paper, we present an empirical analysis of how widely-used general purpose LLMs generate, justify, and rank mobile app recommendations. Our contributions are: (i) a taxonomy of 16 generalizable ranking criteria elicited from LLM outputs; (ii) a systematic evaluation framework to analyse recommendation consistency and responsiveness to explicit ranking instructions; and (iii) a replication package to support reproducibility and future research on AI-based recommendation systems. Our findings reveal that LLMs rely on a broad yet fragmented set of ranking criteria, only partially aligned with standard ASO metrics. While top-ranked apps tend to be consistent across runs, variability increases with ranking depth and search specificity. LLMs exhibit varying sensitivity to explicit ranking instructions - ranging from substantial adaptations to near-identical outputs - highlighting their complex reasoning dynamics in conversational app discovery. Our results aim to support end-users, app developers, and recommender-systems researchers in navigating the emerging landscape of conversational app discovery.
Paper Structure (32 sections, 10 figures, 4 tables)

This paper contains 32 sections, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Mobile App Recommendation
  4. App Store Optimization (ASO)
  5. LLMs as Recommender Systems
  6. Method
  7. Goal and Research Questions
  8. Use Cases
  9. Sample Study: Feature-based Search with Commercial, Web-based LLMs
  10. Dataset
  11. Model selection
  12. Elicitation of Ranking Criteria (RQ1)
  13. Design
  14. ASO Metrics
  15. LLM Ranking Criteria
  16. ...and 17 more sections

Figures (10)

  • Figure 1: Research method design.
  • Figure 2: Design of the criteria elicitation stage (RQ1).
  • Figure 3: Design of the consistency analysis stage (RQ2).
  • Figure 4: Evolution of distinct apps across k positions by model (n=10 runs)
  • Figure 5: App ranking internal consistency heatmaps across different $k$ values using Jaccard similarity (top) and RBO (bottom).
  • ...and 5 more figures