While the capabilities of generative models heavily improved in different domains (images, text, graphs, molecules, etc.), their evaluation metrics largely remain based on simplified quantities or manual inspection with limited practicality. To this end, we propose a framework for Multi-level Performance Evaluation of Generative mOdels (MPEGO), which could be employed across different domains. MPEGO aims to quantify generation performance hierarchically, starting from a sub-feature-based low-level evaluation to a global features-based high-level evaluation. MPEGO offers great customizability as the employed features are entirely user-driven and can thus be highly domain/problem-specific while being arbitrarily complex (e.g., outcomes of experimental procedures). We validate MPEGO using multiple generative models across several datasets from the material discovery domain. An ablation study is conducted to study the plausibility of intermediate steps in MPEGO. Results demonstrate that MPEGO provides a flexible, user-driven, and multi-level evaluation framework, with practical insights on the generation quality. The framework, source code, and experiments will be available at https://github.com/GT4SD/mpego.
翻译:虽然基因模型在不同领域(图像、文本、图表、分子等)的能力大为改善,但其评价指标在很大程度上仍然以简化数量或人工检查为基础,而且实际性有限。为此,我们提议一个可在不同领域使用的对创性模头进行多层次性能评估的框架。MPEGO旨在从基于次级地物的低层次评价到基于全球地貌的高层次评价,按等级量化生成性能。MPEGO提供了很大的定制性能,因为所使用特征完全由用户驱动,因此在任意复杂(例如实验程序的结果)的情况下,可以高度的域/问题特有性。我们确认MPEGO使用多种基因化模型,跨越了材料发现领域的多个数据集。我们进行了一项模拟研究,以研究MPEGO的中间步骤的可取性为起点。结果显示,MPEGOO提供了灵活、用户驱动和多层次的评价框架,对生成质量有实用的洞察力。框架、源代码和实验将在 https://pexmbexgo.