Training neural networks with binary weights and activations is a challenging problem due to the lack of gradients and difficulty of optimization over discrete weights. Many successful experimental results have been achieved with empirical straight-through (ST) approaches, proposing a variety of ad-hoc rules for propagating gradients through non-differentiable activations and updating discrete weights. At the same time, ST methods can be truly derived as estimators in the stochastic binary network (SBN) model with Bernoulli weights. We advance these derivations to a more complete and systematic study. We analyze properties, estimation accuracy, obtain different forms of correct ST estimators for activations and weights, explain existing empirical approaches and their shortcomings, explain how latent weights arise from the mirror descent method when optimizing over probabilities. This allows to reintroduce ST methods, long known empirically, as sound approximations, apply them with clarity and develop further improvements.
翻译:由于缺乏梯度和对离散重量进行优化的困难,培训具有二进制重量和活化作用的神经网络是一个具有挑战性的问题。许多成功的实验成果都是通过实证直通(ST)方法取得的,为通过非差别性活化来传播梯度提出了各种特别规则,并更新了离散重量。与此同时,可以真正将ST方法作为Stochatic二进制网络(SBN)模型中带有Bernoulli重量的估测器。我们将这些推向更完整和系统的研究。我们分析特性、估计准确性、获得不同形式的正确的ST活化和重量估计器、解释现有的实证方法及其缺点、解释在优化概率超过概率时如何从镜子下沉法中产生潜在重量。这样可以重新采用长期已知的ST方法,作为合理的近似值,将这些方法应用得更清晰,并进一步发展。