Due to the surge of cloud-assisted AI services, the problem of designing resilient prediction serving systems that can effectively cope with stragglers/failures and minimize response delays has attracted much interest. The common approach for tackling this problem is replication which assigns the same prediction task to multiple workers. This approach, however, is very inefficient and incurs significant resource overheads. Hence, a learning-based approach known as parity model (ParM) has been recently proposed which learns models that can generate parities for a group of predictions in order to reconstruct the predictions of the slow/failed workers. While this learning-based approach is more resource-efficient than replication, it is tailored to the specific model hosted by the cloud and is particularly suitable for a small number of queries (typically less than four) and tolerating very few (mostly one) number of stragglers. Moreover, ParM does not handle Byzantine adversarial workers. We propose a different approach, named Approximate Coded Inference (ApproxIFER), that does not require training of any parity models, hence it is agnostic to the model hosted by the cloud and can be readily applied to different data domains and model architectures. Compared with earlier works, ApproxIFER can handle a general number of stragglers and scales significantly better with the number of queries. Furthermore, ApproxIFER is robust against Byzantine workers. Our extensive experiments on a large number of datasets and model architectures also show significant accuracy improvement by up to 58% over the parity model approaches.
翻译:由于云层协助的AI服务激增,设计有弹性的预测服务系统以有效应对累赘/故障和尽量减少响应延误的问题引起了很大的兴趣。解决这一问题的共同办法是复制,将同样的预测任务分配给多个工人。然而,这种方法效率非常低,产生大量资源间接费用。因此,最近提议了一种以学习为基础的方法,称为对等模式(ParM),它学习模型,为一组预测产生相似性,以便重建对缓慢/失败工人的预测。虽然这种基于学习的方法比复制更节省资源,但它适合由云体主持的具体模型,特别适合少数的查询(通常少于4个),并能够容忍极少数(最差的一个)施压者数目。此外,ParM并不处理拜占庭制的对抗对抗敌制工人的模式。我们提出了一种不同的方法,名为Apbribeard Compilational Reference(AproproIFER),它不需要在任何平等模型上比复制效率更高的资源效率,但是它特别适合由云层主持的具体模型,因此它可以大量地展示一个更高级的模型。