The early pioneering Neural Architecture Search (NAS) works were multi-trial methods applicable to any general search space. The subsequent works took advantage of the early findings and developed weight-sharing methods that assume a structured search space typically with pre-fixed hyperparameters. Despite the amazing computational efficiency of the weight-sharing NAS algorithms, it is becoming apparent that multi-trial NAS algorithms are also needed for identifying very high-performance architectures, especially when exploring a general search space. In this work, we carefully review the latest multi-trial NAS algorithms and identify the key strategies including Evolutionary Algorithm (EA), Bayesian Optimization (BO), diversification, input and output transformations, and lower fidelity estimation. To accommodate the key strategies into a single framework, we develop B2EA that is a surrogate assisted EA with two BO surrogate models and a mutation step in between. To show that B2EA is robust and efficient, we evaluate three performance metrics over 14 benchmarks with general and cell-based search spaces. Comparisons with state-of-the-art multi-trial algorithms reveal that B2EA is robust and efficient over the 14 benchmarks for three difficulty levels of target performance. The B2EA code is publicly available at \url{https://github.com/snu-adsl/BBEA}.
翻译:早期开创性神经结构搜索(NAS)工程是适用于任何一般搜索空间的多审判方法。随后的工程利用了早期发现和开发的权重共享方法,这些方法假定了结构化搜索空间,通常使用预先固定的超参数。尽管权重共享NAS算算法的计算效率惊人,但越来越明显的是,还需要多审判性NAS算法来确定非常高性能的建筑,特别是在探索一般搜索空间时。在这项工作中,我们仔细审查最新的多审判性NAS算法,并确定了关键战略,包括进化阿尔戈里什姆(EA)、巴耶西亚优化(BO)、多样化、输入和产出转换(BO)、低忠诚度估计。为了将关键战略纳入一个单一的框架,我们开发了B2EA,这是一个辅助性EA,用两个B2替代模型和两步之间的突变。为了显示B2EA是稳健的,我们用普通搜索空间和细胞搜索空间的14个基准,我们评估了3个以上的性能度指标。比较了州-亚瑟-多审的多审算法标准。