The recurrent network architecture is a widely used model in sequence modeling, but its serial dependency hinders the computation parallelization, which makes the operation inefficient. The same problem was encountered in serial adder at the early stage of digital electronics. In this paper, we discuss the similarities between recurrent neural network (RNN) and serial adder. Inspired by carry-lookahead adder, we introduce carry-lookahead module to RNN, which makes it possible for RNN to run in parallel. Then, we design the method of parallel RNN computation, and finally Carry-lookahead RNN (CL-RNN) is proposed. CL-RNN takes advantages in parallelism and flexible receptive field. Through a comprehensive set of tests, we verify that CL-RNN can perform better than existing typical RNNs in sequence modeling tasks which are specially designed for RNNs.
翻译：经常性网络架构是一个广泛使用的序列建模模式,但其序列依赖性阻碍了计算平行化,使得操作效率低下。 在数字电子学的早期阶段,序列添加器也遇到同样的问题。 在本文中,我们讨论了经常性神经网络(RNN)和序列添加器之间的相似性。在随身携带外观添加器的启发下,我们向 RNN 引入了携带光头模块,使得RN 能够平行运行。然后,我们设计了平行的 RNN 计算方法,最后提出了“CL-RNN”(CL-RNN) 。 CL-RNN 在平行和灵活的接受字段中占有优势。通过一套全面的测试,我们核实CL-RNN能够比现有的典型的 RNN 模型任务更好地运行,这些模型是专门为 RNN 设计的。
相关内容
微信扫码咨询专知VIP会员