Recent empirical work shows that inconsistent results based on choice of hyperparameter optimization (HPO) configuration are a widespread problem in ML research. When comparing two algorithms J and K searching one subspace can yield the conclusion that J outperforms K, whereas searching another can entail the opposite. In short, the way we choose hyperparameters can deceive us. We provide a theoretical complement to this prior work, arguing that, to avoid such deception, the process of drawing conclusions from HPO should be made more rigorous. We call this process epistemic hyperparameter optimization (EHPO), and put forth a logical framework to capture its semantics and how it can lead to inconsistent conclusions about performance. Our framework enables us to prove EHPO methods that are guaranteed to be defended against deception, given bounded compute time budget t. We demonstrate our framework's utility by proving and empirically validating a defended variant of random search.
翻译:最近的经验工作表明,基于选择超参数优化(HPO)配置的不一致结果在ML研究中是一个普遍的问题。比较两个算法J和K搜索一个子空间时,可以得出J优于K的结论,而搜索另一个子空间则可能产生相反的结果。简而言之,我们选择超参数的方式可以欺骗我们。我们为先前的这项工作提供了理论补充,认为为了避免这种欺骗,应该使从HPO得出结论的过程更加严格。我们称之为超参数优化(EHPO),并提出了一个逻辑框架,以捕捉其语义学和如何导致关于性能的不一致结论。我们的框架使我们能够证明EHPO方法有保证不受欺骗,因为有约束的计算时间预算t。我们通过证明和实验性地验证随机搜索的防御变体来证明我们的框架的效用。