Modern web services routinely provide REST APIs for clients to access their functionality. These APIs present unique challenges and opportunities for automated testing, driving the recent development of many techniques and tools that generate test cases for API endpoints using various strategies. Understanding how these techniques compare to one another is difficult, as they have been evaluated on different benchmarks and using different metrics. To fill this gap, we performed an empirical study aimed to understand the landscape in automated testing of REST APIs and guide future research in this area. We first identified, through a systematic selection process, a set of 10 state-of-the-art REST API testing tools that included tools developed by both researchers and practitioners. We then applied these tools to a benchmark of 20 real-world open-source RESTful services and analyzed their performance in terms of code coverage achieved and unique failures triggered. This analysis allowed us to identify strengths, weaknesses, and limitations of the tools considered and of their underlying strategies, as well as implications of our findings for future research in this area.
翻译:现代网络服务经常为客户提供REST API 功能,这些API为自动化测试提供了独特的挑战和机遇,推动最近开发了许多技术和工具,利用各种战略为API终点点生成测试案例。了解这些技术如何相互比较是困难的,因为根据不同的基准和不同的指标对这些技术进行了评估。为填补这一空白,我们进行了一项经验研究,目的是了解REST API 自动测试的景观,并指导这方面的未来研究。我们首先通过系统选择过程,确定了一套10种最先进的REST API 测试工具,其中包括研究人员和从业人员开发的工具。然后,我们将这些工具应用于20个真实世界开放源码软件服务的基准,并分析了这些工具在所实现的代码覆盖和所引发的独特失败方面的绩效。这一分析使我们能够查明所考虑的工具及其基本战略的优点、弱点和局限性,以及我们发现的结果对今后该领域研究的影响。