Many recent developments on generative models for natural images have relied on heuristically-motivated metrics that can be easily gamed by memorizing a small sample from the true distribution or training a model directly to improve the metric. In this work, we critically evaluate the gameability of these metrics by designing and deploying a generative modeling competition. Our competition received over 11000 submitted models. The competitiveness between participants allowed us to investigate both intentional and unintentional memorization in generative modeling. To detect intentional memorization, we propose the ``Memorization-Informed Fr\'echet Inception Distance'' (MiFID) as a new memorization-aware metric and design benchmark procedures to ensure that winning submissions made genuine improvements in perceptual quality. Furthermore, we manually inspect the code for the 1000 top-performing models to understand and label different forms of memorization. Our analysis reveals that unintentional memorization is a serious and common issue in popular generative models. The generated images and our memorization labels of those models as well as code to compute MiFID are released to facilitate future studies on benchmarking generative models.
翻译:在这项工作中,我们通过设计和部署基因模型竞争,严格评价这些指标的可选性。我们的竞争得到了11 000多个提交的模型。参与者之间的竞争力使我们得以在基因模型中调查有意和无意的记忆化。为了发现有意的记忆化,我们建议采用“记忆化-成形Fr\'echect Coneption Learth' (MIFID)”作为新的记忆化认知度度度量和设计基准程序,以确保取而代之,能够真正改进感知性质量。此外,我们手动检查1,000个顶级模型的代码,以理解和标注不同的记忆化形式。我们的分析表明,无意记忆化是流行基因模型中一个严重和常见的问题。所生成的图像和我们对这些模型的记忆化标记,以及用于计算MIFID的代码,将促进未来对基准模型的研究。