Binary neural networks (BNNs) have attracted broad research interest due to their efficient storage and computational ability. Nevertheless, a significant challenge of BNNs lies in handling discrete constraints while ensuring bit entropy maximization, which typically makes their weight optimization very difficult. Existing methods relax the learning using the sign function, which simply encodes positive weights into +1s, and -1s otherwise. Alternatively, we formulate an angle alignment objective to constrain the weight binarization to {0,+1} to solve the challenge. In this paper, we show that our weight binarization provides an analytical solution by encoding high-magnitude weights into +1s, and 0s otherwise. Therefore, a high-quality discrete solution is established in a computationally efficient manner without the sign function. We prove that the learned weights of binarized networks roughly follow a Laplacian distribution that does not allow entropy maximization, and further demonstrate that it can be effectively solved by simply removing the $\ell_2$ regularization during network training. Our method, dubbed sign-to-magnitude network binarization (SiMaN), is evaluated on CIFAR-10 and ImageNet, demonstrating its superiority over the sign-based state-of-the-arts. Our source code, experimental settings, training logs and binary models are available at https://github.com/lmbxmu/SiMaN.
翻译:二进制神经网络(BNNs)因其高效的存储和计算能力而吸引了广泛的研究兴趣。然而,BNNs所面临的一项重大挑战在于如何处理离散的制约,同时确保比特放大最大化,这通常使他们的重量优化非常困难。现有的方法会放松使用信号函数的学习,这些符号函数只是将正重编码到+1和 -1 。或者,我们制定了一个角对齐目标,将重量的二进制限制到 {0+1} 以解决挑战。在本文中,我们展示了我们体重的二进制通过将高磁度重量编码到+1 和 0 来提供分析解决方案。因此,一个高质量的离散式解决方案以计算高效的方式建立,而没有信号功能。我们证明,二进制网络所学的重量大致沿着拉巴氏分布线,无法实现最小化,并进一步证明,只要在网络训练中简单地删除$_2美元,就能有效地解决这个问题。我们的方法、调制的符号到磁度网络网络的模型和自动智能网络,正在对我们的图像-10的路径进行测试。