Recurrent neural networks (RNNs) are a powerful approach for time series prediction. However, their performance is strongly affected by their architecture and hyperparameter settings. The architecture optimization of RNNs is a time-consuming task, where the search space is typically a mixture of real, integer and categorical values. To allow for shrinking and expanding the size of the network, the representation of architectures often has a variable length. In this paper, we propose to tackle the architecture optimization problem with a variant of the Bayesian Optimization (BO) algorithm. To reduce the evaluation time of candidate architectures the Mean Absolute Error Random Sampling (MRS), a training-free method to estimate the network performance, is adopted as the objective function for BO. Also, we propose three fixed-length encoding schemes to cope with the variable-length architecture representation. The result is a new perspective on accurate and efficient design of RNNs, that we validate on three problems. Our findings show that 1) the BO algorithm can explore different network architectures using the proposed encoding schemes and successfully designs well-performing architectures, and 2) the optimization time is significantly reduced by using MRS, without compromising the performance as compared to the architectures obtained from the actual training procedure.
翻译:经常性神经网络( RNN) 是时间序列预测的有力方法。 但是,它们的性能受到其架构和超参数设置的强烈影响。 RNN的架构优化是一项耗时的任务,其搜索空间通常是真实、整数和绝对值的混合体。为了能够缩小和扩大网络的规模,结构的表述往往有不同的长度。在本文中,我们提议用巴伊西亚优化算法(Bayesian Optimization)的变式来解决结构优化问题。为了缩短候选人结构的评估时间,将无培训的网络性能估计方法(MRS)作为BO的目标功能。此外,我们提出了三种固定长度的编码计划,以应对可变长结构的表述。结果对RNNs的准确和高效设计提出了新的观点,我们验证了三个问题。我们的调查结果显示,1 BO算法可以利用拟议的编码计划探索不同的网络结构,并成功地设计良好的结构,2 优化时间通过使用MS 来大大缩短,而不会影响实际的学习过程的绩效。