Network quantization aims at reducing bit-widths of weights and/or activations, particularly important for implementing deep neural networks with limited hardware resources. Most methods use the straight-through estimator (STE) to train quantized networks, which avoids a zero-gradient problem by replacing a derivative of a discretizer (i.e., a round function) with that of an identity function. Although quantized networks exploiting the STE have shown decent performance, the STE is sub-optimal in that it simply propagates the same gradient without considering discretization errors between inputs and outputs of the discretizer. In this paper, we propose an element-wise gradient scaling (EWGS), a simple yet effective alternative to the STE, training a quantized network better than the STE in terms of stability and accuracy. Given a gradient of the discretizer output, EWGS adaptively scales up or down each gradient element, and uses the scaled gradient as the one for the discretizer input to train quantized networks via backpropagation. The scaling is performed depending on both the sign of each gradient element and an error between the continuous input and discrete output of the discretizer. We adjust a scaling factor adaptively using Hessian information of a network. We show extensive experimental results on the image classification datasets, including CIFAR-10 and ImageNet, with diverse network architectures under a wide range of bit-width settings, demonstrating the effectiveness of our method.
翻译:网络量化旨在减少重量和(或)激活的比分宽度,这对于实施硬件资源有限的深神经网络特别重要。 多数方法使用直通估计度(STE)来培训量化网络,这避免了零梯度问题,以身份功能取代离散器(即圆函数)的衍生物(即圆函数),避免了零梯度问题。 尽管利用STE的量化网络表现得较好,但STE是次优化的,因为它只是简单地在不考虑离散器投入和输出之间离散错误的情况下传播同一梯度。 在本文件中,我们建议采用元素偏移梯度梯度梯度缩放(EWGS),这是STE的一个简单而有效的替代方法,在稳定性和准确性方面培训一个比STE更好的四分位化网络(即圆函数)衍生物。鉴于离散器输出的梯度梯度,EWGS向上或向下调高,并且使用缩放梯度梯度的梯度,用于通过反调化程序对网络进行配置。 缩放取决于每个梯度的梯度网络的标志, 显示每个梯度输出的深度输出的标志, 和在不断调整的图像结构中, 显示一个不易变频度结构中, 显示一个不易变频度结构中, 显示我们数据的导值的导值的导值结构。