Traditional signal processing methods relying on mathematical data generation models have been cast aside in favour of deep neural networks, which require vast amounts of data. Since the theoretical sample complexity is nearly impossible to evaluate, these amounts of examples are usually estimated with crude rules of thumb. However, these rules only suggest when the networks should work, but do not relate to the traditional methods. In particular, an interesting question is: how much data is required for neural networks to be on par or outperform, if possible, the traditional model-based methods? In this work, we empirically investigate this question in two simple examples, where the data is generated according to precisely defined mathematical models, and where well-understood optimal or state-of-the-art mathematical data-agnostic solutions are known. A first problem is deconvolving one-dimensional Gaussian signals and a second one is estimating a circle's radius and location in random grayscale images of disks. By training various networks, either naive custom designed or well-established ones, with various amounts of training data, we find that networks require tens of thousands of examples in comparison to the traditional methods, whether the networks are trained from scratch or even with transfer-learning or finetuning.
翻译:翻译摘要:
传统的依靠数学数据生成模型的信号处理方法已经被深度神经网络所取代,这些神经网络需要海量数据。由于理论样本复杂度几乎不可能评估,所以这些例子通常是用粗略的经验法则估算的。然而,这些规则只是建议网络应该如何工作,但并不相关传统方法。特别是一个有趣的问题是:神经网络需要多少数据才能与传统的基于模型的方法相当或超越其性能,如果可能的话?在本研究中,我们在两个简单的例子中进行了实证研究,其中数据根据精确定义的数学模型生成,而已知最佳或最先进的数学数据不可知解决方案已知。一个问题是反卷积一维高斯信号,另一个问题是在随机灰度图中估计圆的半径和位置。通过训练各种网络,无论是天真的自定义设计还是已经建立好的网络,以及各种数量的训练数据,我们发现与传统的方法相比,网络需要数万个样本,无论是从头开始训练还是通过传递学习或微调训练。