Commercial ML APIs offered by providers such as Google, Amazon and Microsoft have dramatically simplified ML adoption in many applications. Numerous companies and academics pay to use ML APIs for tasks such as object detection, OCR and sentiment analysis. Different ML APIs tackling the same task can have very heterogeneous performance. Moreover, the ML models underlying the APIs also evolve over time. As ML APIs rapidly become a valuable marketplace and a widespread way to consume machine learning, it is critical to systematically study and compare different APIs with each other and to characterize how APIs change over time. However, this topic is currently underexplored due to the lack of data. In this paper, we present HAPI (History of APIs), a longitudinal dataset of 1,761,417 instances of commercial ML API applications (involving APIs from Amazon, Google, IBM, Microsoft and other providers) across diverse tasks including image tagging, speech recognition and text mining from 2020 to 2022. Each instance consists of a query input for an API (e.g., an image or text) along with the API's output prediction/annotation and confidence scores. HAPI is the first large-scale dataset of ML API usages and is a unique resource for studying ML-as-a-service (MLaaS). As examples of the types of analyses that HAPI enables, we show that ML APIs' performance change substantially over time--several APIs' accuracies dropped on specific benchmark datasets. Even when the API's aggregate performance stays steady, its error modes can shift across different subtypes of data between 2020 and 2022. Such changes can substantially impact the entire analytics pipelines that use some ML API as a component. We further use HAPI to study commercial APIs' performance disparities across demographic subgroups over time. HAPI can stimulate more research in the growing field of MLaaS.
翻译:谷歌、亚马逊和微软等提供商提供的 ML API 大大简化了许多应用程序的 ML 的采用。 许多公司和学术界付钱使用 ML API 进行对象检测、 OCR 和情绪分析等任务。 不同的 ML API 处理同一任务的不同性能非常不一。 此外, 支持 API 的 ML 模式也随着时间而变化。 随着ML API 迅速成为宝贵的市场和广泛使用机器学习的一种广泛方式,必须系统地研究和比较不同的API,并描述一段时间内AIPI 的变化情况。 然而,由于缺少数据,这个话题目前没有得到探讨。 在本文中,我们介绍 MAL API (API 历史), 1,761,417个商业 ML AL API 应用程序的纵向数据集(涉及亚马逊、谷歌、I、IMB、微软和其他供应商的应用程序) 跨越多种任务,包括从2020年到2022年的图像标记、语音识别和文字挖掘。 每个实例包括用于 API (ealalal-alalal AS AS IMI IMI 的图像和图像分析) 的大规模数据流流数据运行中,这期间的动态的动态分析。