Efficiently modeling long-range dependencies is an important goal in sequence modeling. Recently, models using structured state space sequence (S4) layers achieved state-of-the-art performance on many long-range tasks. The S4 layer combines linear state space models (SSMs) with deep learning techniques and leverages the HiPPO framework for online function approximation to achieve high performance. However, this framework led to architectural constraints and computational difficulties that make the S4 approach complicated to understand and implement. We revisit the idea that closely following the HiPPO framework is necessary for high performance. Specifically, we replace the bank of many independent single-input, single-output (SISO) SSMs the S4 layer uses with one multi-input, multi-output (MIMO) SSM with a reduced latent dimension. The reduced latent dimension of the MIMO system allows for the use of efficient parallel scans which simplify the computations required to apply the S5 layer as a sequence-to-sequence transformation. In addition, we initialize the state matrix of the S5 SSM with an approximation to the HiPPO-LegS matrix used by S4's SSMs and show that this serves as an effective initialization for the MIMO setting. S5 matches S4's performance on long-range tasks, including achieving an average of 82.46% on the suite of Long Range Arena benchmarks compared to S4's 80.48% and the best transformer variant's 61.41%.
翻译:高效建模长距离依赖性是序列建模中的一个重要目标。 最近, 使用结构化国家空间序列( S4) 的模型在很多远程任务中取得了最先进的性能。 S4 层将线性国家空间模型(SSMM)与深层学习技术相结合,并利用 HIPPO 框架进行在线功能近似,以达到高性能。 然而, 这个框架导致建筑限制和计算困难,使S4 方法难以理解和实施。 我们再次发现, 高性能需要密切遵循 HIPPO 框架。 具体地说, 我们用一个多投入、 多输出(MIIMO) 的SSM 系统, 将S4 S4 SSM 系统的许多独立的单项、 单输出(SISO) 级数据库替换为一个S4 的系统化系统化模型。 SPPO4 用于S4 SDRioralal 的SDRalization SDRioral, 用于SPO4 SDSA 和SDRioralal 的SDSA的SDRal-L IM 的SDMDMSDMS 和SD SA SA SA 的SA SA AS 的SL SA SA SA SA SA 最高级性标的S- s real 的S- s real 的S- s real 的SDRlal 。 Slal 的SDRal 的SM 。