In recent years, many software engineering researchers have begun to include artifacts alongside their research papers. Ideally, artifacts, including tools, benchmarks, and data, support the dissemination of ideas, provide evidence for research claims, and serve as a starting point for future research. However, in practice, artifacts suffer from a variety of issues that prevent the realization of their full potential. To help the software engineering community realize the full potential of artifacts, we seek to understand the challenges involved in the creation, sharing, and use of artifacts. To that end, we perform a mixed-methods study including a survey of artifacts in software engineering publications, and an online survey of 153 software engineering researchers. By analyzing the perspectives of artifact creators, users, and reviewers, we identify several high-level challenges that affect the quality of artifacts including mismatched expectations between these groups, and a lack of sufficient reward for both creators and reviewers. Using Diffusion of Innovations as an analytical framework, we examine how these challenges relate to one another, and build an understanding of the factors that affect the sharing and success of artifacts. Finally, we make recommendations to improve the quality of artifacts based on our results and existing best practices.
翻译:近年来,许多软件工程研究人员已开始将文物与研究论文一起纳入其中,理想的是,包括工具、基准和数据在内的文物,支持思想的传播,为研究主张提供证据,并成为未来研究的起点。然而,实际上,文物存在各种问题,阻碍充分发挥其潜力。为了帮助软件工程界充分发挥艺术品的潜力,我们力求理解创造、分享和使用艺术品所涉及的挑战。为此,我们开展了混合方法研究,包括对软件工程出版物中的文物进行调查,对153名软件工程研究人员进行在线调查。通过分析艺术品创作者、用户和审查者的观点,我们确定了影响艺术品质量的若干高层次挑战,包括这些群体之间期望不匹配,以及缺乏对创造者和审查者双方的充分奖励。利用创新的注入作为分析框架,我们研究这些挑战如何相互关联,并加深对影响艺术品共享和成功的因素的理解。最后,我们根据我们现有的结果和最佳做法,提出提高艺术品质量的建议。