In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by the connection between geometry of the loss landscape and generalization -- including a generalization bound that we prove here -- we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-{10, 100}, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels.
翻译:在当今严重超度的模型中,培训损失的价值对模型概括能力几乎没有多少保障。事实上,正如通常所做的那样,只有优化培训损失价值,才容易导致不理想的模式质量。受损失地貌的几何与概括性(包括我们在这里证明的概括性约束)之间的联系的驱使,我们引入了一个创新的、有效的程序来同时尽量减少损失价值和损失的锐度。特别是,我们的程序,即 " 锐利-意识最小化(SAM) " (SAM)(SAM)(SAM))(SAM)(SAM)(SAM)(SAM)(SAM)(SAM))(S)(SAM)(SAM)(SAM)(SAM)(S)(SAM)(SAM)(SAM)(S)(SAM(SAM)(SAM)(SL)(SLAx(S)(I-M) ) (SLA(I(S) ) (SQ) (I-A(S) ) (S-Ax) (L) (PL) (P) (L) (S) (L) (ID) (P) (LE) (L) (P) (P) (PL) (L) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (PL) (P) (PL) (P) (P) (P) (P) (P) (PL) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (PL) (P) (L) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P) (P