Binary neural networks (BNNs) have demonstrated their ability to solve complex tasks with comparable accuracy as full-precision deep neural networks (DNNs), while also reducing computational power and storage requirements and increasing the processing speed. These properties make them an attractive alternative for the development and deployment of DNN-based applications in Internet-of-Things (IoT) devices. Despite the recent improvements, they suffer from a fixed and limited compression factor that may result insufficient for certain devices with very limited resources. In this work, we propose sparse binary neural networks (SBNNs), a novel model and training scheme which introduces sparsity in BNNs and a new quantization function for binarizing the network's weights. The proposed SBNN is able to achieve high compression factors and it reduces the number of operations and parameters at inference time. We also provide tools to assist the SBNN design, while respecting hardware resource constraints. We study the generalization properties of our method for different compression factors through a set of experiments on linear and convolutional networks on three datasets. Our experiments confirm that SBNNs can achieve high compression rates, without compromising generalization, while further reducing the operations of BNNs, making SBNNs a viable option for deploying DNNs in cheap, low-cost, limited-resources IoT devices and sensors.
翻译:二线神经网络(BNNs)显示,它们有能力以类似精密深度神经网络(DNNs)的精确度解决复杂的任务,同时降低计算力和存储要求,提高处理速度,这些属性使DNN应用程序在互联网T(IoT)装置中的发展和部署具有吸引力。尽管最近有所改善,它们仍受到固定和有限的压缩因素的影响,这可能导致某些设备的资源非常有限,而某些设备可能不够。在这项工作中,我们提议了稀少的二线神经网络(SBNNs),一种新颖的模式和培训计划,在BNNS中引入松散功能,以及一个新的使网络重量实现二线化的量化功能。提议的SBNNN能够达到高压缩系数,并在推论时间减少操作数量和参数。我们还提供了工具,协助SBNNN,同时通过在三个数据集上对线性与革命网络进行一系列试验,研究我们不同压缩因素的方法的普及性特性。我们的实验证实,SBNNNN能够进一步降低标准,而SNNNERs的低压缩率则可以进一步降低标准。