In adversarial training (AT), the main focus has been the objective and optimizer while the model has been less studied, so that the models being used are still those classic ones in standard training (ST). Classic network architectures (NAs) are generally worse than searched NAs in ST, which should be the same in AT. In this paper, we argue that NA and AT cannot be handled independently, since given a dataset, the optimal NA in ST would be no longer optimal in AT. That being said, AT is time-consuming itself; if we directly search NAs in AT over large search spaces, the computation will be practically infeasible. Thus, we propose a diverse-structured network (DS-Net), to significantly reduce the size of the search space: instead of low-level operations, we only consider predefined atomic blocks, where an atomic block is a time-tested building block like the residual block. There are only a few atomic blocks and thus we can weight all atomic blocks rather than find the best one in a searched block of DS-Net, which is an essential trade-off between exploring diverse structures and exploiting the best structures. Empirical results demonstrate the advantages of DS-Net, i.e., weighting the atomic blocks.
翻译:在对抗性培训(AT)中,主要重点一直是目标和优化,而模型研究较少,因此,正在使用的模型仍然是标准培训(ST)中的经典模型。典型网络结构通常比ST中搜索的NAS更差,在AT中应当相同。在本文中,我们认为NA和AT不能独立处理,因为根据数据集,ST中的最佳NA将不再在AT中处于最佳状态。也就是说,AT本身耗费时间;如果我们直接在AT在大搜索空间中搜索NAS,那么计算将实际上不可行。因此,我们提议建立一个多样化结构网络(DS-Net),以大幅缩小搜索空间的大小:而不是低级操作,我们只考虑预先定义的原子区块,因为原子区块是像残余区块一样经过时间考验的建筑块。只有几个原子区块,因此我们可以权衡所有原子块,而不是在搜索的DS-Net区块中找到最佳的块,这是探索不同结构与利用最佳结构之间的一个重要交换。Empris-SNet的优势。