This paper seeks to answer the following question: \textit{"What can we learn by predicting accuracy?"}. Indeed, classification is one of the most popular tasks in machine learning, and many loss functions have been developed to maximize this non-differentiable objective function. Unlike past work on loss function design, which was guided mainly by intuition and theory before being validated by experimentation, here we propose to approach this problem in the opposite way: we seek to extract knowledge by experimentation. This data-driven approach is similar to that used in physics to discover general laws from data. We used a symbolic regression method to automatically find a mathematical expression highly correlated with a linear classifier's accuracy. The formula discovered on more than 260 datasets of embeddings has a Pearson's correlation of 0.96 and a $r^2$ of 0.93. More interestingly, this formula is highly explainable and confirms insights from various previous papers on loss design. We hope this work will open new perspectives in the search for new heuristics leading to a deeper understanding of machine learning theory.
翻译:本文试图回答以下问题 : \ textit { “ 我们通过预测准确性可以学到什么? ” 。 事实上, 分类是机器学习中最受欢迎的任务之一, 许多损失功能已经开发, 以最大限度地实现这一不可区分的目标功能。 与以往主要以直觉和理论为指导,然后通过实验验证的关于损失函数设计的工作不同, 我们在这里建议以相反的方式处理这一问题: 我们试图通过实验来获取知识。 这种数据驱动的方法与物理学中用来从数据中发现一般法则的方法相似。 我们使用象征性回归方法来自动发现数学表达与线性分类器的精确性高度相关。 在超过260个嵌入数据集中发现的公式的相对值为0. 96 美元 和 0.93 美元 。 更有趣的是, 这个公式非常可解释, 并证实了以前关于损失设计的各种论文的洞察力。 我们希望这项工作将打开新的视角, 以寻找新的超自然论导致更深入了解机器学习理论。