Learned Autoscaling for Cloud Microservices with Multi-Armed Bandits

As cloud applications shift from monoliths to loosely coupled microservices, application developers must decide how many compute resources (e.g., VMs) to give to each microservice within an application. This decision affects both (1) the dollar cost to the application developer and (2) the end-to-end latency perceived by the application user. Today, individual microservices are autoscaled by adding VMs whenever utilization metrics (e.g. CPU, RAM) cross a configurable threshold. Utilization-based autoscaling is simple to understand at the level of an individual microservice. However, an application user's end-to-end latency consists of time spent on multiple microservices and each microservice might need a different threshold to achieve an overall end-to-end latency. Further, thresholds are application and workload dependent. We present COLA, an autoscaler for microservice-based applications. COLA learns autoscaling policies tailored to applications and their workloads and allocates VMs to an application's microservices to meet end-to-end latency targets while minimizing dollar cost. Using 5 open-source applications, we compared COLA to several utilization and machine learning based autoscalers. COLA meets a desired median or tail latency target on 31 of 39 workloads where it provides an average cost reduction of 30.9% over the next cheapest autoscaler. COLA is the most cost effective autoscaling policy for 27 of these 31 workloads. The cost savings from managing a cluster with COLA result in COLA paying for its training cost in a few days.