Asynchronous Load Balancing and Auto-scaling: Mean-Field Limit and Optimal Design

We develop a Markovian framework for load balancing where classical algorithms such as Power-of-$d$ are combined with auto-scaling mechanisms, which allow the net service capacity to scale up or down in response to the current load within the same timescale of job dynamics. Our framework is inspired by serverless platforms such as Knative where servers are software functions that can be flexibly instantiated in milliseconds according to scaling rules defined by the users of the serverless platform. The main question is how to design such scaling rules to minimize user-perceived delay performance while guaranteeing low energy consumption. For the first time, we investigate this problem when the auto-scaling and load balancing processes operate \emph{asynchronously}, as in Knative. One advantage induced by asynchronism is that jobs do not necessarily need to wait any time a scale-up decision is taken. In our main result, we find a general condition on the structure of scaling rules able to drive mean-field dynamics to delay and relative energy optimality, i.e., a situation where both the user-perceived delay and the relative energy wastage induced by idle servers vanish in the limit where the network demand grows to infinity in proportion to the nominal service capacity. The identified condition suggests to scale up the current net capacity if and only if the mean demand exceeds the rate at which servers become idle and active. Finally, we propose \emph{Rate-Idle}, i.e., a scaling rule that satisfies our optimality condition, and by means of numerical simulations, we show that it improves delay performance over existing (synchronous) schemes.