The ever-growing computational demands of increasingly complex machine learning models frequently necessitate the use of powerful cloud-based infrastructure for their training. Binary neural networks are known to be promising candidates for on-device inference due to their extreme compute and memory savings over higher-precision alternatives. However, their existing training methods require the concurrent storage of high-precision activations for all layers, generally making learning on memory-constrained devices infeasible. In this article, we demonstrate that the backward propagation operations needed for binary neural network training are strongly robust to quantization, thereby making on-the-edge learning with modern models a practical proposition. We introduce a low-cost binary neural network training strategy exhibiting sizable memory footprint reductions while inducing little to no accuracy loss vs Courbariaux & Bengio's standard approach. These decreases are primarily enabled through the retention of activations exclusively in binary format. Against the latter algorithm, our drop-in replacement sees memory requirement reductions of 3--5$\times$, while reaching similar test accuracy in comparable time, across a range of small-scale models trained to classify popular datasets. We also demonstrate from-scratch ImageNet training of binarized ResNet-18, achieving a 3.78$\times$ memory reduction. Our work is open-source, and includes the Raspberry Pi-targeted prototype we used to verify our modeled memory decreases and capture the associated energy drops. Such savings will allow for unnecessary cloud offloading to be avoided, reducing latency, increasing energy efficiency, and safeguarding end-user privacy.
翻译:日益复杂的机器学习模式日益增长的计算需求往往需要使用强大的云基基础设施来进行培训。二线神经网络由于在高精度替代品方面极端的计算和记忆节约,因此被认为是有希望的在线电导候选人。然而,它们现有的培训方法要求同时储存所有层面的高精度启动功能,通常使得在记忆控制装置上学习不可行。在文章中,我们表明,二线神经网络培训所需的后退传播操作非常强大,足以进行量化,从而使与现代模型的前沿学习成为实用的建议。我们引入了低成本双线网络培训战略,表明记忆足迹减少幅度很大,同时几乎没有造成准确性损失,而Courbariaux和Bengio的标准方法则要求同时保存高精度启动功能,这主要是通过完全以二进制格式保留启动装置来实现。在后一种算法中,我们采用后一种模式替换的存储存储需求将减少3-5美元,同时在可比较的时间实现类似的测试精度精确度,在一系列小型的存储足迹上,我们所培训的存储模型将逐渐减少。