Weight sharing, as an approach to speed up architecture performance estimation has received wide attention. Instead of training each architecture separately, weight sharing builds a supernet that assembles all the architectures as its submodels. However, there has been debate over whether the NAS process actually benefits from weight sharing, due to the gap between supernet optimization and the objective of NAS. To further understand the effect of weight sharing on NAS, we conduct a comprehensive analysis on five search spaces, including NAS-Bench-101, NAS-Bench-201, DARTS-CIFAR10, DARTS-PTB, and ProxylessNAS. We find that weight sharing works well on some search spaces but fails on others. Taking a step forward, we further identified biases accounting for such phenomenon and the capacity of weight sharing. Our work is expected to inspire future NAS researchers to better leverage the power of weight sharing.
翻译:作为加快建筑绩效估计的一种方法,体重共享得到了广泛的关注。作为加快建筑绩效估计的一种方法,重量共享不是单独培训每个建筑,而是建立一个将所有建筑集合成其子型的超级网。然而,由于超级网络优化与NAS的目标之间存在差距,对NAS进程是否真正受益于重量共享进行了辩论。为了进一步理解重量共享对NAS的影响,我们对五个搜索空间进行了全面分析,包括NAS-Bench-101、NAS-Bench-201、NASS-Bench-201、DARSS-CIFAR10、DARSS-PTB和ProoxlessNAS。我们发现,重量共享在某些搜索空间运作良好,但在另一些搜索空间则失败。我们进一步确定了对这种现象和重量共享能力的偏差。我们的工作有望激励未来的NAS研究人员更好地利用重量共享的力量。