Human-in-the-loop optimization utilizes human expertise to guide machine optimizers iteratively and search for an optimal solution in a solution space. While prior empirical studies mainly investigated novices, we analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We conducted a study (N=60) in text, photo, and 3D mesh optimization contexts. We found that novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization iteration with more explicit preference while keeping satisfaction low. In contrast, novices were more easily satisfied and terminated faster. Therefore, we identified that experts seek more diverse outcomes while the machine reaches optimal results, and the observed behavior can be used as a performance indicator for human-in-the-loop system designers to improve underlying models. We inform future research to be cautious about the impact of user expertise when designing human-in-the-loop systems.
翻译:人类在环形优化利用人的专门知识来引导机器优化器的迭接和在解决方案空间寻找最佳解决方案。 虽然先前的经验研究主要调查新手,但我们分析了专门知识水平对结果质量和相应的主观满意度的影响。 我们在文本、照片和3D网状优化背景下进行了一项研究(N=60)。我们发现,新手可以达到专家质量业绩水平,但拥有较高专门知识的参与者导致更优化的迭代,更明确偏好,同时保持低的满意度。相比之下,新手更容易满足并更快地终止。因此,我们发现专家在机器取得最佳结果时寻求更多不同的结果,所观察到的行为可以用作动态系统设计者改进基本模型的绩效指标。我们告知未来研究,在设计“在环形人”系统时,要谨慎对待用户专门知识的影响。