Significant progress has been achieved in automating the design of various components in deep networks. However, the automatic design of loss functions for generic tasks with various evaluation metrics remains under-investigated. Previous works on handcrafting loss functions heavily rely on human expertise, which limits their extendibility. Meanwhile, existing efforts on searching loss functions mainly focus on specific tasks and particular metrics, with task-specific heuristics. Whether such works can be extended to generic tasks is not verified and questionable. In this paper, we propose AutoLoss-Zero, the first general framework for searching loss functions from scratch for generic tasks. Specifically, we design an elementary search space composed only of primitive mathematical operators to accommodate the heterogeneous tasks and evaluation metrics. A variant of the evolutionary algorithm is employed to discover loss functions in the elementary search space. A loss-rejection protocol and a gradient-equivalence-check strategy are developed so as to improve the search efficiency, which are applicable to generic tasks. Extensive experiments on various computer vision tasks demonstrate that our searched loss functions are on par with or superior to existing loss functions, which generalize well to different datasets and networks. Code shall be released.
翻译:在设计深层网络各组成部分方面已取得重大进展。然而,对各种评价指标通用任务损失功能的自动设计仍未得到充分调查。以前关于手工艺损失功能的工程严重依赖人的专门知识,这限制了其扩展性。与此同时,现有的寻找损失功能的工作主要侧重于具体任务和特定指标,并带有任务特有的杂质。这种工程能否扩大到通用任务,是无法核实和值得怀疑的。在本文件中,我们提议AutoLos-Zero,这是从头到尾查找损失功能以完成通用任务的第一个总框架。具体地说,我们设计了一个基本搜索空间,仅由原始数学操作员组成,以适应不同的任务和评估指标。采用演进算法的一种变式,以发现初级搜索空间的损失功能。为了提高搜索效率,将适用于通用任务,将制定损失反馈协议和梯度等值检查战略。关于各种计算机视觉任务的广泛实验表明,我们所搜索的损失功能与现有损失功能相同或优于现有损失功能,这些功能一般地适用于不同的数据集和网络。