We present unit scaling, a paradigm for designing deep learning models that simplifies the use of low-precision number formats. Training in FP16 or the recently proposed FP8 formats offers substantial efficiency gains, but can lack sufficient range for out-of-the-box training. Unit scaling addresses this by introducing a principled approach to model numerics: seeking unit variance of all weights, activations and gradients at initialisation. Unlike alternative methods, this approach neither requires multiple training runs to find a suitable scale nor has significant computational overhead. We demonstrate the efficacy of unit scaling across a range of models and optimisers. We further show that existing models can be adapted to be unit-scaled, training BERT-Large in FP16 and then FP8 with no degradation in accuracy.
翻译:我们提出了单元缩放方法,这是一种用于设计深度学习模型的范式,可以简化低精度数值格式的使用。在FP16或最近提出的FP8格式中进行训练可以带来实质性的效率提升,但可能缺乏足够的训练范围。单元缩放通过引入一种原则性方法来解决这个问题:在初始化时寻求所有权重、激活和梯度的单位方差。与替代方法不同,这种方法既不需要多次运行找到合适的尺度,也没有显著的计算开销。我们在各种模型和优化器上展示了单元缩放的功效。我们进一步展示,现有模型可以适应为单元缩放模型,使用FP16训练BERT-Large,然后使用FP8进行训练,精度没有降低。