Beyond Relevance: An Adaptive Exploration-Based Framework for Personalized Recommendations

TL;DR

This paper tackles the problem of balancing personalization, diversity, and cold-start robustness in dynamic recommendation environments. It introduces an adaptive exploration based framework that uses sentence-transformer embeddings and an online adaptive clustering with dynamic thresholding, coupled with a user controlled exploration mechanism to promote novelty while preserving relevance. The authors validate the approach on MovieLens and through LLM-based simulated A/B testing, showing reduced intra-list similarity and increased unexpectedness, along with strong user preference for exploratory recommendations among long-term users. Computational analyses indicate linear scaling with the number of clusters and practical efficiency for moderate-scale deployments, highlighting the method as a viable path to mitigating over-specialization in content recommendations while maintaining personalization and scalability.

Abstract

Recommender systems must balance personalization, diversity, and robustness to cold-start scenarios to remain effective in dynamic content environments. This paper introduces an adaptive, exploration-based recommendation framework that adjusts to evolving user preferences and content distributions to promote diversity and novelty without compromising relevance. The system represents items using sentence-transformer embeddings and organizes them into semantically coherent clusters through an online algorithm with adaptive thresholding. A user-controlled exploration mechanism enhances diversity by selectively sampling from under-explored clusters. Experiments on the MovieLens dataset show that enabling exploration reduces intra-list similarity from 0.34 to 0.26 and increases unexpectedness to 0.73, outperforming collaborative filtering and popularity-based baselines. A/B testing with 300 simulated users reveals a strong link between interaction history and preference for diversity, with 72.7% of long-term users favoring exploratory recommendations. Computational analysis confirms that clustering and recommendation processes scale linearly with the number of clusters. These results demonstrate that adaptive exploration effectively mitigates over-specialization while preserving personalization and efficiency.