With modern requirements, there is an increasing tendency of considering multiple objectives/criteria simultaneously in many Software Engineering (SE) scenarios. Such a multi-objective optimization scenario comes with an important issue -- how to evaluate the outcome of optimization algorithms, which typically is a set of incomparable solutions (i.e., being Pareto non-dominated to each other). This issue can be challenging for the SE community, particularly for practitioners of Search-Based SE (SBSE). On one hand, multi-objective optimization could still be relatively new to SE/SBSE researchers, who may not be able to identify the right evaluation methods for their problems. On the other hand, simply following the evaluation methods for general multi-objective optimization problems may not be appropriate for specific SE problems, especially when the problem nature or decision maker's preferences are explicitly/implicitly available. This has been well echoed in the literature by various inappropriate/inadequate selection and inaccurate/misleading use of evaluation methods. In this paper, we first carry out a systematic and critical review of quality evaluation for multi-objective optimization in SBSE. We survey 717 papers published between 2009 and 2019 from 36 venues in seven repositories, and select 95 prominent studies, through which we identify five important but overlooked issues in the area. We then conduct an in-depth analysis of quality evaluation indicators/methods and general situations in SBSE, which, together with the identified issues, enables us to codify a methodological guidance for selecting and using evaluation methods in different SBSE scenarios.
翻译:随着现代要求的出现,在许多软件工程(SE)情景中,越来越倾向于同时考虑多重目标/标准。这种多目标优化设想方案涉及一个重要问题 -- -- 如何评价优化算法的结果,优化算法通常是一套无法比较的解决办法(即,Pareto不以彼此为主),这个问题对SE社区,特别是搜索SE(SBSS)的从业人员来说可能具有挑战性。一方面,多目标优化对于SE/SBSE研究人员来说可能仍然比较新,他们可能无法找到解决他们问题的适当评价方法。另一方面,仅仅采用一般多目标优化算法的评估方法,通常不是适合具体的SE问题,特别是当问题性质或决策者的偏好明确/不明显可得时。 这个问题在文献中得到了很好的响应,特别是对于SBSS(SSSS)的实践者来说,多目标化评估方法评估可能比较新,我们首先对SBSE(SBE)中多目标优化的质量评估方法进行了系统和批判性审查。 另一方面,我们在2009年和2019年出版的论文中,我们从一个重要的方法分析中,从SBS(我们从一个重要)到在7个地点,从SBS/CR(我们选择了一个重要)的文献中选择了一个重要的方法分析中, 和205个研究中,我们从SBS(我们从一个重要)到一个重要)的分类(我们从一个研究中选择了一个重要)到一个重要的方法,在5个研究中选择了一种方法,在5个研究中选择了一个重要)的系列中选择了一种方法。