In this paper, we propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost. We first construct a baseline network by modifying and binarizing a compact real-valued network with parameter-free shortcuts, bypassing all the intermediate convolutional layers including the downsampling layers. This baseline network strikes a good trade-off between accuracy and efficiency, achieving superior performance than most of existing binary networks at approximately half of the computational cost. Through extensive experiments and analysis, we observed that the performance of binary networks is sensitive to activation distribution variations. Based on this important observation, we propose to generalize the traditional Sign and PReLU functions, denoted as RSign and RPReLU for the respective generalized functions, to enable explicit learning of the distribution reshape and shift at near-zero extra cost. Lastly, we adopt a distributional loss to further enforce the binary network to learn similar output distributions as those of a real-valued network. We show that after incorporating all these ideas, the proposed ReActNet outperforms all the state-of-the-arts by a large margin. Specifically, it outperforms Real-to-Binary Net and MeliusNet29 by 4.0% and 3.6% respectively for the top-1 accuracy and also reduces the gap to its real-valued counterpart to within 3.0% top-1 accuracy on ImageNet dataset.
翻译:在本文中,我们提出若干加强二进制网络的想法,以便在不产生任何额外的计算费用的情况下,消除实际价值网络的准确性差距。我们首先通过修改和采用无参数捷径,绕过包括下层抽样层在内的所有中间进化层,绕过所有中间进化层,绕过所有中间进化层,包括下层抽样层,从而在准确性与效率之间作出良好的权衡,以大约计算成本的一半计算出优于现有多数现有二进制网络。通过广泛的实验和分析,我们发现二进制网络的性能对激活分布变异非常敏感。根据这一重要观察,我们提议将传统的Sign和PRELU功能普遍化为无参数实际价值网络和PRPRELU, 从而能够明确了解分布的重塑和转换,以近零的额外费用。最后,我们采用分配损失的方法进一步实施二进制网络,以学习与实际价值网络类似的产出分布。我们发现,在纳入所有这些想法后,拟议的ReActNet将所有真实值比值比值比值为3,我们提议将所有真实的Sign-imal-%和Mel-1.ximal 4,并分别将真实地将真实的图像比值比值比值比值从正正正正正正值为3.和最大比值最高比值为最高比值。