As the state-of-the-art machine learning methods in many fields rely on larger datasets, storing datasets and training models on them become significantly more expensive. This paper proposes a training set synthesis technique for data-efficient learning, called Dataset Condensation, that learns to condense large dataset into a small set of informative synthetic samples for training deep neural networks from scratch. We formulate this goal as a gradient matching problem between the gradients of deep neural network weights that are trained on the original and our synthetic data. We rigorously evaluate its performance in several computer vision benchmarks and demonstrate that it significantly outperforms the state-of-the-art methods. Finally we explore the use of our method in continual learning and neural architecture search and report promising gains when limited memory and computations are available.
翻译:由于许多领域最先进的机器学习方法依赖较大的数据集,储存数据集和培训模型的费用大大增加。本文件建议采用一套培训综合技术,用于数据高效学习,称为 " 数据集集中 ",该技术学会从头到尾将大型数据集凝结成一小套信息丰富的合成样本,用于培训深神经网络。我们将此目标作为一个梯度,匹配在原始和合成数据方面受过训练的深神经网络重量梯度之间的问题。我们严格评价其几个计算机愿景基准的性能,并表明它大大超过最新方法。最后,我们探索了如何利用我们的方法,从头到尾不断学习和神经结构搜索,并在有有限的记忆和计算时报告有希望的成果。