Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.
翻译：在许多人工智能领域,例如自然语言处理、计算机视觉和音频处理领域,变异器取得了巨大成功,因此,自然会吸引学术和产业研究人员的大量兴趣。但到目前为止,已经建议了各种各样的变异器(a.k.a.X-exists),但是,对这些变异器的系统和全面的文献审查仍然缺失。在这次调查中,我们提供了对各种变异器的全面审查。我们首先简要地介绍了香草变异器,然后提出了新的X-exists分类法。接下来,我们从三个角度介绍各种变异器:建筑改造、培训前和应用。最后,我们概述了未来研究的一些潜在方向。
相关内容
微信扫码咨询专知VIP会员