Neural Architecture Search (NAS) has gained significant popularity as an effective tool for designing high performance deep neural networks (DNNs). NAS can be performed via policy gradient, evolutionary algorithms, differentiable architecture search or tree-search methods. While significant progress has been made for both policy gradient and differentiable architecture search, tree-search methods have so far failed to achieve comparable accuracy or search efficiency. In this paper, we formulate NAS as a Combinatorial Multi-Armed Bandit (CMAB) problem (CMAB-NAS). This allows the decomposition of a large search space into smaller blocks where tree-search methods can be applied more effectively and efficiently. We further leverage a tree-based method called Nested Monte-Carlo Search to tackle the CMAB-NAS problem. On CIFAR-10, our approach discovers a cell structure that achieves a low error rate that is comparable to the state-of-the-art, using only 0.58 GPU days, which is 20 times faster than current tree-search methods. Moreover, the discovered structure transfers well to large-scale datasets such as ImageNet.
翻译:神经结构搜索(NAS)作为设计高性能深神经网络(DNNS)的有效工具,已获得显著支持。NAS可以通过政策梯度、进化算法、不同的建筑搜索或树木搜索方法进行。虽然在政策梯度和不同的建筑搜索方面取得了显著进展,但树搜索方法迄今未能达到可比的准确性或搜索效率。在本文中,我们将NAS作为一个多武装混合匪帮问题(CMAB-NAS)进行设计。这样就可以将大型搜索空间分解成小块,使树木搜索方法能够更有效、更高效地应用。我们进一步利用以树为基础的方法解决CMAB-NAS问题。在CIFAR-10中,我们的方法发现一个低误差率的单元格结构,与最新工艺相近,仅使用0.58个GPU日,比目前的树搜索方法快20倍。此外,发现的结构向图像网络等大型数据集转移了很多。