Multi-label classification tasks such as OCR and multi-object recognition are a major focus of the growing machine learning as a service industry. While many multi-label prediction APIs are available, it is challenging for users to decide which API to use for their own data and budget, due to the heterogeneity in those APIs' price and performance. Recent work shows how to select from single-label prediction APIs. However the computation complexity of the previous approach is exponential in the number of labels and hence is not suitable for settings like OCR. In this work, we propose FrugalMCT, a principled framework that adaptively selects the APIs to use for different data in an online fashion while respecting user's budget. The API selection problem is cast as an integer linear program, which we show has a special structure that we leverage to develop an efficient online API selector with strong performance guarantees. We conduct systematic experiments using ML APIs from Google, Microsoft, Amazon, IBM, Tencent and other providers for tasks including multi-label image classification, scene text recognition and named entity recognition. Across diverse tasks, FrugalMCT can achieve over 90% cost reduction while matching the accuracy of the best single API, or up to 8% better accuracy while matching the best API's cost.
翻译:多标签分类任务,例如 OCR 和多球识别等,是不断增长的机器学习作为服务行业的主要焦点。虽然有许多多标签预测API,但用户很难决定自己的数据和预算使用哪一种API,原因是这些API的价格和性能各不相同。最近的工作表明如何从单标签预测API中选择。然而,先前方法的计算复杂性在标签数量上是指数指数指数指数的指数,因此不适合OCR等环境。在这项工作中,我们提议FrugalMCT,这是一个原则性框架,在尊重用户预算的同时,适应性地选择API用于在线方式的不同数据。API选择问题是一个整数线性程序,我们展示了一个特殊结构,我们利用它来开发一个高效的在线API选择器,并有很强的绩效保证。我们利用谷歌、微软、亚马逊、IBM、Tencent和其他供应商对包括多标签图像分类、现场文本识别和命名实体识别等任务进行系统实验。在降低最佳成本的同时,将ABIMCT与最佳的准确性匹配。