In this paper, we present connections between three models used in different research fields: weighted finite automata~(WFA) from formal languages and linguistics, recurrent neural networks used in machine learning, and tensor networks which encompasses a set of optimization techniques for high-order tensors used in quantum physics and numerical analysis. We first present an intrinsic relation between WFA and the tensor train decomposition, a particular form of tensor network. This relation allows us to exhibit a novel low rank structure of the Hankel matrix of a function computed by a WFA and to design an efficient spectral learning algorithm leveraging this structure to scale the algorithm up to very large Hankel matrices.We then unravel a fundamental connection between WFA and second-orderrecurrent neural networks~(2-RNN): in the case of sequences of discrete symbols, WFA and 2-RNN with linear activationfunctions are expressively equivalent. Leveraging this equivalence result combined with the classical spectral learning algorithm for weighted automata, we introduce the first provable learning algorithm for linear 2-RNN defined over sequences of continuous input vectors.This algorithm relies on estimating low rank sub-blocks of the Hankel tensor, from which the parameters of a linear 2-RNN can be provably recovered. The performances of the proposed learning algorithm are assessed in a simulation study on both synthetic and real-world data.
翻译:在本文中,我们展示了不同研究领域使用的三种模型之间的联系:来自正式语言和语言的加权限量自动数据~(WFA),机器学习中使用的经常性神经网络,以及包含量子物理和数字分析中使用的高阶气压优化技术集集集的强调网络。我们首先展示了WFA和高压火车分解(一种特殊的发声网络形式)之间的内在关系。这种关系使我们能够展示汉克尔矩阵中由WFA计算函数的新颖的低级结构,并设计一种高效的光谱学习算法,利用这一结构将算法提升到非常大的汉克尔矩阵。然后我们揭示了WFA和二级经常神经网络(2-RNNN)之间的一种基本连接:在离散符号序列中,WFA和2-RNNN和线性激活功能的分解序列中,我们展示了这种等同结果,同时将经典光谱学习算算法用于加权自动数据,我们引入了第一个用于连续输入矢量矢量序列的线性2-RNNNE的光学算法,这个算法可以依据对低级的合成标准进行模拟分析。在模拟研究中,在Sqrml-rsal-sal-simal-simal-simal-assimal-assimal imalalalalalalalalisalisalisalisalisalisalisalisalisalisalisalisalisalisalisalisalisalisalisal sual sual sual sual suvalisal suvalisal sual labal subal subal subal subal subal subal subal subal subal subal sal sual subal ladal subal subal sal lad subal subaldal subal ladal subal subal subal sual ladal ladal ladal labal ladal labaldaldaldal ladal lax中,从Sal la