Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation

Recently, automatic configuration tuning has attracted intensive interests from the database community to improve the performance of modern database management systems (DBMS). As a result, a line of configuration tuning systems with new algorithms and advanced features have been developed. However, the existing evaluations of database tuning systems are conducted system-wide in limited scenarios. It remains unclear to identify the best combination of algorithm designs for database configuration tuning in practice, given the large body of solutions and the expensive evaluations. In addition, when jumping out of the database community, we could find even more underlying solutions that are designed for configuration tuning. To this end, this paper provides a comprehensive evaluation of configuration tuning techniques from a broader domain, not limited to database community. In particular, we present a unified pipeline of database knob tuning with three key components and evaluate the fine-grained intra-algorithms in various and challenging scenarios. Our evaluation has demonstrated that the hype-parameter optimization techniques can be borrowed to facilitate the database configuration tuning and we further identify the best solution ``path'' in different scenarios. We identify design trade-offs to suggest desirable optimizations and directions for future development of database tuning approaches. Beyond the comprehensive evaluations, we offer an efficient and unified benchmark via surrogate that reduces the evaluation cost to a minimum, allowing for extensive runs and analysis of new optimizers.