Modern ML methods excel when training data is IID, large-scale, and well labeled. Learning in less ideal conditions remains an open challenge. The sub-fields of few-shot, continual, transfer, and representation learning have made substantial strides in learning under adverse conditions; each affording distinct advantages through methods and insights. These methods address different challenges such as data arriving sequentially or scarce training examples, however often the difficult conditions an ML system will face over its lifetime cannot be anticipated prior to deployment. Therefore, general ML systems which can handle the many challenges of learning in practical settings are needed. To foster research towards the goal of general ML methods, we introduce a new unified evaluation framework - FLUID (Flexible Sequential Data). FLUID integrates the objectives of few-shot, continual, transfer, and representation learning while enabling comparison and integration of techniques across these subfields. In FLUID, a learner faces a stream of data and must make sequential predictions while choosing how to update itself, adapt quickly to novel classes, and deal with changing data distributions; while accounting for the total amount of compute. We conduct experiments on a broad set of methods which shed new insight on the advantages and limitations of current solutions and indicate new research problems to solve. As a starting point towards more general methods, we present two new baselines which outperform other evaluated methods on FLUID. Project page: https://raivn.cs.washington.edu/projects/FLUID/.
翻译:现代ML方法在培训数据为 IID 、 大型和标签良好的培训数据时优于现代 ML 方法。 在不理想条件下学习仍然是一项公开的挑战。 少发、连续、转移和代表性学习的子领域在不利条件下的学习中取得了长足的进步;每个领域都通过方法和洞察力提供了独特的优势。这些方法解决了不同的挑战,如按顺序或稀缺的培训实例提供的数据,然而,通常在部署之前无法预见ML系统在一生中将面临的困难条件。因此,需要通用的 ML系统来处理在实际环境中学习的许多挑战。为了促进对一般 ML方法目标的研究,我们引入了新的统一评价框架-FLUID(灵活序列数据) 。FLUID整合了少发、连续、转移和代表性学习的目标,同时使这些技术能够在这些子领域进行比较和整合。在FLUID之前,学习者将面临数据流流,必须作出连续的预测,同时选择如何更新自己,迅速适应新的课程,并处理不断变化的数据分配;同时计算出新的数据分配情况,同时计算出新的ILD/ LFLFL方法的总规模,我们开始的深度分析方法。