Continuous-depth neural models, where the derivative of the model's hidden state is defined by a neural network, have enabled strong sequential data processing capabilities. However, these models rely on advanced numerical differential equation (DE) solvers resulting in a significant overhead both in terms of computational cost and model complexity. In this paper, we present a new family of models, termed Closed-form Continuous-depth (CfC) networks, that are simple to describe and at least one order of magnitude faster while exhibiting equally strong modeling abilities compared to their ODE-based counterparts. The models are hereby derived from the analytical closed-form solution of an expressive subset of time-continuous models, thus alleviating the need for complex DE solvers all together. In our experimental evaluations, we demonstrate that CfC networks outperform advanced, recurrent models over a diverse set of time-series prediction tasks, including those with long-term dependencies and irregularly sampled data. We believe our findings open new opportunities to train and deploy rich, continuous neural models in resource-constrained settings, which demand both performance and efficiency.
翻译:由神经网络来界定模型隐藏状态衍生物的连续深度神经模型,通过这种模型的衍生物由神经网络来界定,因此能够产生强大的连续处理数据的能力;然而,这些模型依靠先进的数字差异方程式(DE)解答器,从而在计算成本和模型复杂性两方面都产生了巨大的间接费用。在本文中,我们提出了一套新的模型,称为封闭式连续深度网络(CfC),这些模型简单易描述,而且至少一个数量级更快,同时展示出与其基于 ODE 的对应方相比同样强大的建模能力。这些模型由此产生出自一组明确的时间持续模型的分析封闭式解决方案,从而减轻了对复杂的DE 解析器的需求。在我们的实验性评估中,我们证明CfC 网络在一系列不同的时间序列预测任务中,包括长期依赖性和不定期抽样的数据,超越了先进的经常性模型。我们认为,我们的调查结果为在资源紧张的环境中培训和部署丰富、连续的神经模型开辟了新的机会,这要求既具有性又具有效率。