A Critical Study on Data Leakage in Recommender System Offline Evaluation

In academic research, recommender models are often evaluated on offline datasets. The offline dataset is first split to training and test instances. All training instances are then modelled in a user-item interaction matrix which can be used to train recommender models. Many such offline evaluations ignore the global timeline in the data, which leads to "data leakage": a model learns from future data to predict a current value, making the evaluation unrealistic. In this paper, we evaluate the impact of "data leakage" using two widely adopted baseline models, BPR and NeuMF, on four popular offline datasets - MovieLens-25M, Yelp, Amazon-music, and Amazon-electronic. We show that accessing to different amount of future data may improve or deteriorate a model's recommendation accuracy. That is, ignoring global timeline in offline evaluation makes the performance among recommendation models not comparable. We share our understanding of these observations and highlight the importance of preserving the global timeline. We also call for a revisit of recommender system offline evaluation.