This paper analyzes three formal models of Transformer encoders that differ in the form of their self-attention mechanism: unique hard attention (UHAT); generalized unique hard attention (GUHAT), which generalizes UHAT; and averaging hard attention (AHAT). We show that UHAT and GUHAT Transformers, viewed as string acceptors, can only recognize formal languages in the complexity class AC$^0$, the class of languages recognizable by families of Boolean circuits of constant depth and polynomial size. This upper bound subsumes Hahn's (2020) results that GUHAT cannot recognize the DYCK languages or the PARITY language, since those languages are outside AC$^0$ (Furst et al., 1984). In contrast, the non-AC$^0$ languages MAJORITY and DYCK-1 are recognizable by AHAT networks, implying that AHAT can recognize languages that UHAT and GUHAT cannot.
翻译:本文分析了三种以自我注意机制形式不同的变换器编码器的正式模式:独特的难感(UHAT);普遍独特的难感(GUHAT),它一般地概括了UHAT;以及平均地难感(AHAT)。我们表明,UHAT和GUHAT变换器,作为接受字符串的接受者,只能承认复杂等级AC$0的正规语言,即由持续深度和多面体大小的波林电路家族所识别的一类语言。这一上界次组合(2020年)的结果是,GUHAT无法承认DYCK语言或种族语言,因为这些语言在AC$0(Furst等人,1984年)之外。相比之下,非AC$0美元语言MAJORITY和DYCK-1可以被AHAT网络所识别,意味着AHAT可以识别UHAT和GUHAT无法识别的语言。