Network binarization emerges as one of the most promising compression approaches offering extraordinary computation and memory savings by minimizing the bit-width. However, recent research has shown that applying existing binarization algorithms to diverse tasks, architectures, and hardware in realistic scenarios is still not straightforward. Common challenges of binarization, such as accuracy degradation and efficiency limitation, suggest that its attributes are not fully understood. To close this gap, we present BiBench, a rigorously designed benchmark with in-depth analysis for network binarization. We first carefully scrutinize the requirements of binarization in the actual production and define evaluation tracks and metrics for a comprehensive and fair investigation. Then, we evaluate and analyze a series of milestone binarization algorithms that function at the operator level and with extensive influence. Our benchmark reveals that 1) the binarized operator has a crucial impact on the performance and deployability of binarized networks; 2) the accuracy of binarization varies significantly across different learning tasks and neural architectures; 3) binarization has demonstrated promising efficiency potential on edge devices despite the limited hardware support. The results and analysis also lead to a promising paradigm for accurate and efficient binarization. We believe that BiBench will contribute to the broader adoption of binarization and serve as a foundation for future research.
翻译:网络的二进制作为最有希望的压缩方法之一,通过将比重最小化,实现特殊的计算和记忆节约。然而,最近的研究表明,在现实情景下,将现有的二进制算法应用于各种任务、架构和硬件,仍然不是直截了当的。双进制的共同挑战,如精度退化和效率限制,表明其属性没有得到完全理解。为了缩小这一差距,我们提出BiBench,这是对网络二进制进行深入分析的严格设计的基准。我们首先仔细审查实际生产中的二进制要求,并为全面和公正的调查确定评估轨道和衡量标准。然后,我们评估和分析一系列具有里程碑意义的双进制算法,这些算法在操作者一级运作并具有广泛影响。我们的基准表明,1)二进制操作者对双进制网络的性能和可部署性具有关键影响;2)二进制的准确性因不同的学习任务和神经结构而大不相同;3)二进制表明,尽管硬件支持有限,但边缘设备的效率潜力大不小。结果和分析也导致一个有希望的范例,在准确和高效的双进制基础基础化方面进行更广泛的研究。