Deep Reinforcement Learning for Orchestrating Cost-Aware Reconfigurations of vRANs

Virtualized Radio Access Networks (vRANs) are fully configurable and can be implemented at low cost over commodity platforms offering unprecedented network management flexibility. In this paper, a novel vRAN reconfiguration problem with a deep reinforcement learning (RL)-based framework is proposed that jointly reconfigures the functional splits of the base stations (BSs), resources and locations of the virtualized central units (vCUs) and distributed units (vDUs), and the routing for each BS flow. The objective is to minimize the long-term total network operation cost while adapting to traffic demands and resource availability. Testbed measurements are performed to study the relations between traffic demands and computing resource utilization, which reveal that their relations have high variance and dependence on platform and platform load. Hence, acquiring the perfect model of the underlying system is non-trivial. A comprehensive cost function is formulated that considers resource overprovisioning, instantiation and reconfiguration and the declined demands, where such impacts urge to perform the reconfigurations prudently. Motivated by these insights, our solution framework is developed using model-free RL. Since the formulated RL problem has a semi-continuous state and discrete action space, Dueling Double Deep Q-network (D3QN)-based approach is proposed. However, the system consists of multiple BSs with highly coupled configurations sharing the same resources, which renders a multi-dimensional discrete action space and drives combinatorial growth of the number of possible actions. To overcome the curse of dimensionality, action branching, an action decomposition method with a shared decision module followed by neural network branches, is tailored with D3QN. Further, simulations are performed using O-RAN compliant model and real traces of the testbed.