Recurrent Neural Networks (RNNs) have been widely used in sequence analysis and modeling. However, when processing high-dimensional data, RNNs typically require very large model sizes, thereby bringing a series of deployment challenges. Although various prior works have been proposed to reduce the RNN model sizes, executing RNN models in resource-restricted environments is still a very challenging problem. In this paper, we propose to develop extremely compact RNN models with fully decomposed hierarchical Tucker (FDHT) structure. The HT decomposition does not only provide much higher storage cost reduction than the other tensor decomposition approaches but also brings better accuracy performance improvement for the compact RNN models. Meanwhile, unlike the existing tensor decomposition-based methods that can only decompose the input-to-hidden layer of RNNs, our proposed fully decomposition approach enables the comprehensive compression for the entire RNN models with maintaining very high accuracy. Our experimental results on several popular video recognition datasets show that our proposed fully decomposed hierarchical tucker-based LSTM (FDHT-LSTM) is extremely compact and highly efficient. To the best of our knowledge, FDHT-LSTM, for the first time, consistently achieves very high accuracy with only few thousand parameters (3,132 to 8,808) on different datasets. Compared with the state-of-the-art compressed RNN models, such as TT-LSTM, TR-LSTM and BT-LSTM, our FDHT-LSTM simultaneously enjoys both order-of-magnitude (3,985x to 10,711x) fewer parameters and significant accuracy improvement (0.6% to 12.7%).
翻译:经常性神经网络(RNN)已被广泛用于序列分析和建模。然而,当处理高维数据时,RNN通常需要非常大的模型规模,从而带来一系列部署挑战。尽管已经提出了各种先前的工程来缩小 RNN 模型的大小,但在资源限制环境中执行 RNN 模型仍然是一个非常具有挑战性的问题。在本文件中,我们提议开发非常紧凑的 RNN 模型,具有完全分解的等级Tuck(FDHT)结构。HT的分解不仅比其他高分辨率解构方法多得多的存储成本削减,而且为NNNNNNM模型带来更好的精确性能改进。同时,与现有的高压分解RNNNS模型输入到高层次层相比,我们拟议的完全压缩整个 RNNNM模型的精度和完全分解的等级LSTTM(FDT-NLT-NLT-NLT-NLT) 的实验结果显示,我们提议的完全分解的级LSTM(FT-NLT-NLT-NLT-NLM-NLM-NLT-M-M-M-M-M-M-M-M-M-T-M-ID-ID-T-T-T-NL-ID-T-T-T-T-T-T-T-T-T-T-T-ID-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T-T