Conducting efficient performance estimations of neural architectures is a major challenge in neural architecture search (NAS). To reduce the architecture training costs in NAS, one-shot estimators (OSEs) amortize the architecture training costs by sharing the parameters of one supernet between all architectures. Recently, zero-shot estimators (ZSEs) that involve no training are proposed to further reduce the architecture evaluation cost. Despite the high efficiency of these estimators, the quality of such estimations has not been thoroughly studied. In this paper, we conduct an extensive and organized assessment of OSEs and ZSEs on three NAS benchmarks: NAS-Bench-101/201/301. Specifically, we employ a set of NAS-oriented criteria to study the behavior of OSEs and ZSEs and reveal that they have certain biases and variances. After analyzing how and why the OSE estimations are unsatisfying, we explore how to mitigate the correlation gap of OSEs from several perspectives. For ZSEs, we find that current ZSEs are not satisfying enough in these benchmark search spaces, and analyze their biases. Through our analysis, we give out suggestions for future application and development of efficient architecture performance estimators. Furthermore, the analysis framework proposed in our work could be utilized in future research to give a more comprehensive understanding of newly designed architecture performance estimators. All codes and analysis scripts are available at https://github.com/walkerning/aw_nas.
翻译:对神经结构进行高效的性能估计是神经结构搜索(NAS)的一大挑战。为了降低NAS的建筑培训成本,所有结构之间共享一个超级网的参数,对建筑培训成本进行一次性估算(OSEs),最近,为进一步降低建筑评估成本,提出了没有经过任何培训的零射线估测(ZSeses),以进一步降低建筑评估成本。尽管这些估测者的效率很高,但此类估算的质量还没有得到彻底研究。在本文中,我们根据三个NAS基准(NAS-Bench-101/201/201/301)对OSS和ZESE进行了广泛而有组织的评估。具体地说,我们使用一套面向NAS的建筑培训标准来研究OSes和ZSE的行为,并表明它们存在某些偏差和差异。在分析OSEOSE估计如何和为什么不令人满意之后,我们探索如何从几个角度来缩小OSESE的相互关联差距。关于ZSeses的我们发现,当前的ZSeses没有在三个基准搜索空间和今后分析中充分满足了我们提出的业绩分析。我们未来分析的系统分析。我们未来分析中,我们用所有分析的系统分析是用来进行新的分析。