The pre-training of masked language models (MLMs) consumes massive computation to achieve good results on downstream NLP tasks, resulting in a large carbon footprint. In the vanilla MLM, the virtual tokens, [MASK]s, act as placeholders and gather the contextualized information from unmasked tokens to restore the corrupted information. It raises the question of whether we can append [MASK]s at a later layer, to reduce the sequence length for earlier layers and make the pre-training more efficient. We show: (1) [MASK]s can indeed be appended at a later layer, being disentangled from the word embedding; (2) The gathering of contextualized information from unmasked tokens can be conducted with a few layers. By further increasing the masking rate from 15% to 50%, we can pre-train RoBERTa-base and RoBERTa-large from scratch with only 78% and 68% of the original computational budget without any degradation on the GLUE benchmark. When pre-training with the original budget, our method outperforms RoBERTa for 6 out of 8 GLUE tasks, on average by 0.4%.
翻译:蒙面语言模型(MLMS) 的预培训需要大量计算,才能在下游NLP任务上取得良好结果,从而产生巨大的碳足迹。 在香草 MLM 中, 虚拟标牌、 [MASK] 充当占位符, 从无包装的标牌中收集背景化信息, 以恢复腐败信息。 这提出了我们是否可以在后一层附加[MASK], 以缩短前层的序列长度, 提高培训前的效率 。 我们显示:(1) [MASK] 确实可以在后一层附加, 与嵌入的字脱钩; (2) 从无包装标牌收集背景化信息可以用几层进行。 通过进一步将遮盖率从15%提高到50%, 我们可以将RoBERTA基地和RoBERTA(ROBERTA) 从头开始, 最初的计算预算只有78%和68%, 而不会在GLUE基准上出现任何退化 。 我们显示:(1) 在原始预算培训前, 我们的方法在8 GLUE任务中平均0.4% 6比ROBERTA 。