Online crowdsourcing platforms make it easy to perform evaluations of algorithm outputs with surveys that ask questions like "which image is better, A or B?") The proliferation of these "user studies" in vision and graphics research papers has led to an increase of hastily conducted studies that are sloppy and uninformative at best, and potentially harmful and misleading. We argue that more attention needs to be paid to both the design and reporting of user studies in computer vision and graphics papers. In an attempt to improve practitioners' knowledge and increase the trustworthiness and replicability of user studies, we provide an overview of methodologies from user experience research (UXR), human-computer interaction (HCI), and related fields. We discuss foundational user research methods (e.g., needfinding) that are presently underutilized in computer vision and graphics research, but can provide valuable guidance for research projects. We provide further pointers to the literature for readers interested in exploring other UXR methodologies. Finally, we describe broader open issues and recommendations for the research community. We encourage authors and reviewers alike to recognize that not every research contribution requires a user study, and that having no study at all is better than having a carelessly conducted one.
翻译:在线众包平台使得通过调查对算法产出进行评估变得容易,这些调查提出了诸如“哪一种图像更好,A或B”这样的问题。 )这些“用户研究”在视觉和图形研究论文中扩散,导致匆忙进行的研究增加,这些研究充其量是草率的,没有信息规范,而且可能有害和误导。我们争辩说,需要更多地注意计算机视觉和图形文件中用户研究的设计与报告。为了提高从业人员的知识,提高用户研究的可信度和可复制性,我们从用户经验研究(UXR)、人类-计算机互动(HCI)和相关领域中提供了方法概览。我们讨论了基本用户研究方法(例如需要调查),这些方法目前在计算机视觉和图形研究中没有得到充分利用,但可以为研究项目提供宝贵的指导。我们为有兴趣探索其他UXR方法的读者进一步提供文献提示。最后,我们向研究界介绍更广泛的公开问题和建议。我们鼓励作者和评论员们认识到,并非每一项研究贡献都需要用户研究,而且没有任何研究的改进。