The effectiveness of machine learning algorithms arises from being able to extract useful features from large amounts of data. As model and dataset sizes increase, dataset distillation methods that compress large datasets into significantly smaller yet highly performant ones will become valuable in terms of training efficiency and useful feature extraction. To that end, we apply a novel distributed kernel based meta-learning framework to achieve state-of-the-art results for dataset distillation using infinitely wide convolutional neural networks. For instance, using only 10 datapoints (0.02% of original dataset), we obtain over 65% test accuracy on CIFAR-10 image classification task, a dramatic improvement over the previous best test accuracy of 40%. Our state-of-the-art results extend across many other settings for MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and SVHN. Furthermore, we perform some preliminary analyses of our distilled datasets to shed light on how they differ from naturally occurring data.
翻译:机器学习算法的效力来自于能够从大量数据中提取有用的特征。随着模型和数据集规模的增加,将大型数据集压缩成大小得多但性能强的数据集蒸馏方法在培训效率和有用特性提取方面将变得宝贵。为此,我们运用一个以内部内核为基础的新的分布式元学习框架,利用无限宽广的进化神经网络实现数据蒸馏的最先进的结果。例如,我们仅使用10个数据点(占原始数据集的0.02%),就CIFAR-10图像分类任务获得了65%的测试精度,大大改进了以前40%的最佳测试精度。我们对MNIST、Fashion-MNIST、CIFAR-10、CIFAR-100和SVHN的许多其他环境都采用了我们最新的数据元件进行了一些初步分析,以说明它们与自然数据有何不同。