As machine learning permeates more industries and models become more expensive and time consuming to train, the need for efficient automated hyperparameter optimization (HPO) has never been more pressing. Multi-step planning based approaches to hyperparameter optimization promise improved efficiency over myopic alternatives by more effectively balancing out exploration and exploitation. However, the potential of these approaches has not been fully realized due to their technical complexity and computational intensity. In this work, we leverage recent advances in Transformer-based, natural-language-interfaced hyperparameter optimization to circumvent these barriers. We build on top of the recently proposed OptFormer which casts both hyperparameter suggestion and target function approximation as autoregressive generation thus making planning via rollouts simple and efficient. We conduct extensive exploration of different strategies for performing multi-step planning on top of the OptFormer model to highlight its potential for use in constructing non-myopic HPO strategies.
翻译:由于机器学习渗透到更多的工业和模型中,而且培训耗费的时间越来越昂贵,因此,现在比以往任何时候都更迫切需要高效的自动化超参数优化(HPO),以多步骤规划为基础的超参数优化方法通过更有效地平衡勘探和开发,有望提高超光谱优化替代方法的效率;然而,由于这些方法的技术复杂性和计算强度,这些方法的潜力尚未充分实现。在这项工作中,我们利用基于变异器的、以自然语言为界面的超光谱优化的最新进展来绕过这些障碍。我们利用最近提出的“奥普福德”方案,它既提出了超光谱仪建议,又将目标功能近似为自动递增生成,从而使通过推出方案进行规划变得简单高效。我们广泛探索了在“奥普福德”模型上进行多步规划的不同战略,以突出其在构建非流星式HPO战略方面的潜力。