Software crowdsourcing platforms employ extrinsic rewards such as rating or ranking systems to motivate workers. Such rating systems are noisy and provide limited knowledge about workers' preferences and performance. To develop better understanding of worker reliability and trustworthiness in software crowdsourcing, this paper reports an empirical study conducted on more than one year's real-world data from TopCoder, one of the leading software crowdsourcing platforms. To do so, first, we create a bipartite network of active workers based on common task registrations. Then, we use the Clauset-Newman-Moore graph clustering algorithm to identify worker clusters in the network. Finally, we conduct an empirical evaluation to measure and analyze workers' behavior per identified community in the platform by workers' rating. More specifically, workers' behavior is analyzed based on their performances in terms of reliability, trustworthiness, and success; their preferences in terms of efficiency and elasticity; and strategies in terms of comfort, confidence, and deceitfulness. The main result of this study identified four communities of active workers: mixed-ranked, high-ranked, mid-ranked, and low-ranked. This study shows that the low-ranked community associates with the highest reliable workers with an average reliability of 25%, while the mixed-ranked community contains the most trustworthy workers with average trustworthiness of 16%. Such empirical evidence is beneficial to help exploring resourcing options while understanding the relations among unknown resources to improve task success.
翻译:为了更好地了解工人的可靠性和在软件众包中的信任度,本文报告了对来自软件众包平台之一TopCoder的一年以上真实世界数据进行的实证研究。为此,首先,我们根据共同任务登记,建立一个双方活跃工人网络,如评级或排名制度来激励工人。然后,我们使用Clatter-Newman-Moore图表群集算法来识别网络中的工人集群。最后,我们进行了实证评估,通过工人的评级来衡量和分析平台中每个已查明社区的工人行为。更具体地说,根据工人行为分析的依据是他们在可靠性、可信度和成功性方面的表现;他们在效率和弹性方面的偏好;以及基于共同任务登记的战略。我们首先,我们创建了一个双方活跃工人网络。我们通过这一研究的主要结果确定了四个积极的工人群体:混合级别、高级别、中级别和低级别。这项研究显示,工人的行为依据的是他们在可靠性、信任度和16个最低水平的行业群落的可靠程度,同时展示了他们与16个行业之间的可靠程度。