Nobody of the Crowd: An Empirical Evaluation on Worker Clustering in Topcoder

Context: Software crowdsourcing platforms typically employ extrinsic rewards such as rating or ranking systems to motivate workers. Such rating systems are noisy and only provide limited knowledge about worker's preferences and performance. Goal: The objective of this study is to empirically investigate patterns and effects of worker behavior in software crowdsourcing platforms in order to improve the success and efficiency of software crowdsourcing. Method: First, we create the bipartite network of active workers based on common registration for tasks. Then, we use the Clauset-Newman-Moore graph clustering algorithm to identify developer clusters in the network. Finally, we conduct an empirical evaluation to measure and analyze workers' behavior per identified cluster in the platform by workers' ranking. More specifically, workers' behavior is analyzed based on worker reliability, worker trustworthiness, and worker success as measures for workers' performance, worker efficiency, and worker elasticity to represent workers' preferences, and worker contest, worker confidence, and worker deceitfulness to understand workers' strategies. The empirical study is conducted on more than one year's real-world data from topcoder, one of the leading software crowdsourcing platforms. Results: We identify four clusters of active workers: mixed-ranked, high-ranked, mid-ranked, and low-ranked. Based on statistical analysis, this study can only support that the low ranked group associates with the highest reliable workers with an average reliability of 25%, while the mixed-ranked group contains the most trustworthy workers with average trustworthiness of 16%. Conclusions: These findings are helpful for task requesters to understand preferences and relations among unknown resources in the platform and plan for task success in a more effective and efficient manner in a software crowdsourcing platform.