Inspired by the human cognitive system, attention is a mechanism that imitates the human cognitive awareness about specific information, amplifying critical details to focus more on the essential aspects of data. Deep learning has employed attention to boost performance for many applications. Interestingly, the same attention design can suit processing different data modalities and can easily be incorporated into large networks. Furthermore, multiple complementary attention mechanisms can be incorporated in one network. Hence, attention techniques have become extremely attractive. However, the literature lacks a comprehensive survey specific to attention techniques to guide researchers in employing attention in their deep models. Note that, besides being demanding in terms of training data and computational resources, transformers only cover a single category in self-attention out of the many categories available. We fill this gap and provide an in-depth survey of 50 attention techniques categorizing them by their most prominent features. We initiate our discussion by introducing the fundamental concepts behind the success of attention mechanism. Next, we furnish some essentials such as the strengths and limitations of each attention category, describe their fundamental building blocks, basic formulations with primary usage, and applications specifically for computer vision. We also discuss the challenges and open questions related to attention mechanism in general. Finally, we recommend possible future research directions for deep attention.
翻译:在人类认知系统的启发下,关注是一种机制,它模仿了人类对特定信息的认知意识,扩大了关键细节,以更加侧重于数据的基本方面。深层学习利用了注意力来提高许多应用的性能。有趣的是,同样的关注设计可以适应不同的数据模式的处理,并且容易纳入大型网络。此外,多种互补关注机制可以被纳入一个网络。因此,关注技术变得极具吸引力。然而,文献缺乏一项全面的调查,具体用于指导研究人员在深层模型中利用关注的注意技术。注意到变压器除了要求培训数据和计算资源之外,还只覆盖了现有许多类别中的单一自省类别。我们填补了这一空白,对50种按其最突出特征分类的注意技术进行了深入的调查。我们开始讨论,引入了关注成功机制背后的基本概念。接着,我们提供了一些基本要点,如每个关注类别的长处和局限性,描述其基本构件、基本配有基本用途和专门用于计算机愿景的应用。我们还讨论了与关注机制有关的挑战和开放问题和一般研究方向。最后,建议我们有可能提出未来关注。